{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Functions\n",
    "\n",
    "When fitting data to labels we need to measure the degree of goodness of fit. This sounds obvious but isn't quite so straightforward. In fact, there are entire fields of statistics that focus solely on that (e.g. robust statistics). In this notebook we'll discuss a number of ways how we can measure whether our model is doing well. As a side-benefit, we'll get to know the loss function layers in ``gluon``. We begin with our default import ritual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import mxnet.gluon as gluon\n",
    "from mxnet import nd, autograd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import mxnet.autograd as ag\n",
    "mx.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression - L1 loss\n",
    "\n",
    "As we discussed in the introduction, regression describes the cases where we want to estimate some real valued number $f(x) \\in \\mathbb{R}$ to match an observation $y$. A natural idea of measuring the distance would be to compute $|y - f(x)|$. This makes sense, e.g. if we need to estimate how much it might cost to manufacture a product: if we estimate too low, we will incur a loss due to underestimation. If we overprice it, we will sell fewer products (here we're making the unrealistic assumption that both are equally bad). In math, the loss function is\n",
    "\n",
    "$$l(y,f) = |y-f|$$\n",
    "\n",
    "Let's compute it with ``gluon`` and also its gradient. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4lfX9//HnJ8nJJAsIM2xkZpKlRUBx4UJAURkhCUHU\nWhd10x+OYqvVVqtSEQlJCKK4cBSEOlCg2kzC3jvMBDJJQtbn9wckX1tBMs459xnvx3VxCeHkc78O\nHF/c5x7vo7TWCCGEcBwuRgcQQghhXlLsQgjhYKTYhRDCwUixCyGEg5FiF0IIByPFLoQQDkaKXQgh\nHIwUuxBCOBgpdiGEcDBuRmy0Y8eOunfv3kZsWggh7FZubm6R1jroUo8zpNh79+5NTk6OEZsWQgi7\npZQ62JzHyaEYIYRwMFLsQgjhYKTYhRDCwUixCyGEg5FiF0IIB2OWq2KUUgeAcqAeqNNaR5tjXSGE\nEC1nzssdr9ZaF5lxPSGEEK1gV4di8g+XMP+HvUbHEEKIVsncdwprfBypuYpdA/9SSuUqpWZe6AFK\nqZlKqRylVE5hYWGrNvJpXgEvfbWD1H/vb0tWIYSwutR/7+euBf9hxeZjFt+WuQ7FXKm1PqKU6gR8\nrZTaobVe+/MHaK0XAAsAoqOjW/VP1pxbhnC8tJrnv9xGgLeJ8ZHBbU8uhBAWtnxDAc9/uY3rh3Rm\nzNAuFt+eWfbYtdZHzv/3JLAciDXHuv/LzdWFNyZF8pt+HXjso018u/2EJTYjhBBm892OEzz20SZ+\n068Db0yKxM3V8kfA27wFpZSPUsq38efA9cCWtq57MZ4mVxZMi2ZoNz9++14emftOWWpTQgjRJln7\nT3P/kjyGdvNjwbRoPE2uVtmuOf7p6AysV0ptBLKAFVrrVWZY96LaebiRmhhD90AvZqTnsPVoqSU3\nJ4QQLbb1aCnJadl0D/QiLSmWdh7Wm7nY5mLXWu/TWoef/zFUa/2iOYJdSod2HixJjsPX042ERVns\nLzpjjc0KIcQl7S86Q8KiLHw93ViSHEd7H3erbt+uLnf8X90CvMiYEUeDhqkLMzleWm10JCGEkzte\nWs3UhZk0aMiYEUe3AC+rZ7DrYgfoF9SO9KRYSipriE/JpPhMjdGRhBBOqqSyhmmLMimtqiU9KZZ+\nQe0MyWH3xQ4QGuzPuwnRHDxdSVJaNmfO1hkdSQjhZM6crSMxNZsDpyp5d1o0ocH+hmVxiGIH+E2/\njrw5KZJNBSXctySXs3X1RkcSQjiJs3X13Lckl00FJbw5KZIr+nUwNI/DFDvADUO78PLtYazbXcSs\nZRupb7D8rbtCCOdW36CZtWwj63YX8fLtYdxghRuQLsWQzzy1pInRPSitqmXuiu34ebnxp/GhKKWM\njiWEcEBaa/7w2RZWbD7GH24ezMToHkZHAhyw2AFmjOjL6TM1/OP7vQR6u/PEmEFGRxJCOKBXVu/k\n/axDPHB1P2aM6Gt0nCYOWewAj98wkJKq2qZyv2ek7fyhCyHs37tr9/GP7/cyOa4nj10/0Og4/8Vh\ni10pxR9vC6G0qpYXV27H39vEnTbyNkkIYd8+zDnMiyu3c3NYV/54W4jNHe512GIHcHVRvHZnBGVV\ntTz1ySb8PE2MCTH+xIYQwn6t3nqcpz7ZxIjLOvLanRG4uthWqYODXRVzIe5uLsyfGkV4jwAeen8D\nP+6VD3kSQrTOj3uLeHDpBsJ7BDB/ahTubrZZobaZysx8zg8N693Rm3vSc9hUUGJ0JCGEndlUUMI9\n6Tn07uhNamIMPlYc6tVSTlHsAAHe7iyeHkegjzuJqdnsOVlhdCQhhJ3Yc7KCxNRsAn3cyUiOI8Db\nukO9Wsppih2gi78nGclxuCiIT8nkSEmV0ZGEEDbuSEkV01IycVGwJDmOzn6eRke6JKcqdoA+HX1I\nnx5LRXUd8SmZnKo4a3QkIYSNOlVxlviUTMqr60ifHkvvjj5GR2oWpyt2gKHd/ElJjOFIcRWJqdmU\nV9caHUkIYWMqzg/1OlJcRUpiDEO7GTfUq6WcstgBYvu05+2pw9h2rIyZi3OprpWhYUKIc6pr67kn\nPYftx8p4e+owYvu0NzpSizhtsQOMHtSZv04M56d9p3jw/Q3U1TcYHUkIYbC6+gYeen8DP+07xasT\nwxk9qLPRkVrMqYsdYFxkd567dQhfbzvBU59uRmuZCCmEs9Ja8/Snm/nXthM8d+sQxkV2NzpSq9ju\nhZhWlDi8D8WVtfz9290Eept45qbBNneLsBDCsrTW/Gnldj7KLeDhay4jcXgfoyO1mhT7eY9cexkl\nlTW8u24/Ad7uPHB1f6MjCSGs6O0f9vLuuv0kXNGLR669zOg4bSLFfp5SimdvHUpJVS2vrN5JoLc7\nk+N6Gh1LCGEFSzMP8ZdVO7ktohvP3jrU7t+xS7H/jIuL4tWJ4ZRV1TL7s834e5m4Oayr0bGEEBa0\ncvMxZn+2masHBvHqxHBcbHCoV0s5/cnT/2VydeEfU6KI7hXII8s2sHZXodGRhBAWsm53IQ9/sIHo\nXoH8Y0oUJlfHqESzPQullKtSaoNS6p/mWtMoXu6uLEyIoV9QO+7NyCXvULHRkYQQZrbhUDH3ZuTS\nL6gdCxNi8HJ3NTqS2Zjzn6eHge1mXM9Q/l4mFifH0snPg6TUbHadKDc6khDCTHadKCcxNZsgXw8W\nJ8fi72UyOpJZmaXYlVLBwM3AQnOsZys6+XqyJDkODzcX4lMyOXy60uhIQog2Ony6kviUTDzcXFiS\nHEcnX9sf6tVS5tpjfx14ArjorZtKqZlKqRylVE5hof0ct+7R3puM5DiqaxuIT8mksFyGhglhrwrL\nzw31qq5tICM5jh7tvY2OZBFtLnal1C3ASa117q89Tmu9QGsdrbWODgoKautmrWpgF18WJcZwouws\n0xZlUVolQ8OEsDdl1bUkLMriRNlZFiXGMLCLr9GRLMYce+zDgbFKqQPAB8BopdQSM6xrU6J6BTI/\nPoo9J8uZkZ5NVY0MDRPCXlTV1DMjLYfdJ8uZHx9FVK9AoyNZVJuLXWv9tNY6WGvdG7gb+E5rPbXN\nyWzQqAFB/O3OCHIOFvPA0jxqZWiYEDavtr6B3y3NI/vgaf52ZwSjBtjXEYPWcIyLNq3o1vBuzB0X\nwnc7TvL4RxtpaJChYULYqoYGzRMfb+LbHSeZOy6EW8O7GR3JKsx656nW+nvge3OuaYumxPWipPLc\n6IEAb3eevXWI3d+CLISj0Vrzwj+3sXzDER6/YSBT4noZHclqZKRAK/32qn6cPlNDyvr9BHq787Cd\nDw0SwtG88e0e0n48QPKVffjtVf2MjmNVUuytpJRi9k2DKams5bVvdhHgbSLhN72NjiWEABb/dIDX\nvtnF7cOCme2EY7il2NvAxUXx8u2hlFXX8uwXWwnwNnFbhH0O5hfCUXyef4Rnv9jKdUM68/LtoQ4x\n1Kul5ORpG7m5uvDmpEji+rTn9x9uZM2Ok0ZHEsJprdlxkt9/uJHY3u15c1Ikbg4y1KulnPNZm5mn\nyZWFCdEM6urLfUtyyT5w2uhIQjid7AOnuf+9XAZ19WVhQjSeJscZ6tVSUuxm4utpIi0plu4BXkxP\ny2bb0TKjIwnhNLYfK2N6Wjbd/L1IS4rF19Oxhnq1lBS7GXVs50HGjDjaebgxbVEWB4rOGB1JCId3\n8NQZ4lOyaOfhRsaMODq28zA6kuGk2M2se4AXGcmx1Dc0MDUlkxNl1UZHEsJhnSirZmpKJvUNDWQk\nn3vHLKTYLaJ/J1/SkmIpPlPDtJQsSiprjI4khMMpraxlWkoWpytqSEuKpX8nxx3q1VJS7BYS3iOA\nBdOi2V90hulp2VTW1BkdSQiHUVlTR1JaFvuLzrBgWjThPQKMjmRTpNgtaHj/jrwxKYL8wyXcm5FL\nTZ0MDROirWrqGrhvSR75h0t4Y1IEw/t3NDqSzZFit7AxIV3584RQ1u0uYtaH+dTL0DAhWq2+QTPr\nw3zW7irkzxNCGRPS1ehINknuPLWCu2J6UlJZy5+/2oG/l4m540Kc7hZnIdpKa82cz7fwz03HePrG\nQdwV09PoSDZLit1K7h3Vj+LKWub/sJdAb3ceu2Gg0ZGEsCt/+3oX72Ue4r5R/bh3lHMN9WopKXYr\nenLMQEoqa3hrzR4CvE3MGNHX6EhC2IWU9ft587s93B3TgyfHyE7RpUixW5FSihfHh1JaVcvcFdsJ\n8Hbnjqhgo2MJYdM+yS3gj//cxo0hXXhxfKgcxmwGOXlqZa4uitfvjuDK/h158pNNfL3thNGRhLBZ\nX287wROfbOLK/h15/e4IXJ1wUmNrSLEbwMPNlXfiowjp7s8DS/P4ae8poyMJYXP+s+8UDyzNI6S7\nP+/ER+Hh5rxDvVpKit0gPh5upCXG0LO9N/cszmFzQanRkYSwGVuOlDIjPYee7b1JS4zBx0OOGreE\nFLuBAn3cyUiOxd/LREJqFnsLK4yOJITh9hVWkLAoC38vExnJsQT6uBsdye5IsRusq/+5oWEKmJaS\nxdGSKqMjCWGYY6VVxKdkAZCRHEtXfxnq1RpS7Dagb1A70qfHUlZVS3xKJqfPyNAw4XxOn6khPiWL\nsqpa0qfH0jeondGR7JYUu40I6e7PwoRoCoqrSEzNouKsDA0TzqPibB1JqVkcPl3JuwnRhHT3NzqS\nXWtzsSulPJVSWUqpjUqprUqp580RzBnF9e3AvMnD2Hq0jJmLczhbV290JCEs7mxdPfdm5LDlaBnz\nJg/j8r4djI5k98yxx34WGK21DgcigDFKqcvNsK5TunZIZ165I4wf957i4ffzqauXiZDCcdU3aB5+\nP59/7znFK3eEce2QzkZHcghtLnZ9TuPlHKbzP2SEYRtMGBbMnFuGsGrrcWYv34LW8scpHI/Wmmc+\n3cyqrceZc8sQJgyTu7DNxSwXhyqlXIFcoD8wT2udaY51ndn0K/tQUlnDG9+dmyvz9E2DjY4khFm9\ntGoHy3IO8+Do/ky/so/RcRyKWYpda10PRCilAoDlSqkQrfWWnz9GKTUTmAnQs6eM22yOR68bQHFl\nLe+s3Uegjzv3yUQ74SDm/7CXd37YR/zlvZh13QCj4zgcs14Vo7UuAdYAYy7wewu01tFa6+igoCBz\nbtZhKaV4fuxQbg3vxktf7eCDrENGRxKizZZlH+Klr3Zwa3g3nh87VIZ6WYA5rooJOr+njlLKC7gO\n2NHWdcU5Li6Kv04MZ9SAIJ5ZvpmvNh8zOpIQrbZqyzGe/nQzowYE8deJ4bjIUC+LMMcee1dgjVJq\nE5ANfK21/qcZ1hXnubu5MH9qFJE9A3n4g3zW7y4yOpIQLbZ+dxEPvZ9PZM9A3p46DHc3uY3GUsxx\nVcwmrXWk1jpMax2itX7BHMHEf/Nyd2VRQgx9g3yYmZFD/uESoyMJ0Wz5h0uYmZFD3yAfFiXE4O0u\nQ70sSf7JtCP+3iYWT4+lYzsPElOz2H2i3OhIQlzSnpPlJKVm0bGdB4unx+LvbTI6ksOTYrcznfw8\nWZIch8nVhfiULAqKK42OJMRFFRRXMnVhFm6uLixJjqOTn6fRkZyCFLsd6tnBm8XTY6msqSM+JYui\nirNGRxLiF4oqzhKfkkVlTR2Lp8fSs4O30ZGchhS7nRrc1Y9FiTEcK60iYVEWZdW1RkcSokl5dS0J\ni7I4VlrFosQYBnf1MzqSU5Fit2PRvdszf2oUO4+XMyM9h+paGRomjFddW8+M9Bx2Hi/n7alRRPdu\nb3QkpyPFbueuGtiJv90VQfaB0/xuaR61MjRMGKiuvoHfLd1A1oHT/PXOcK4e2MnoSE5Jit0BjA3v\nxgu3hfDN9pM8+fEmGhpkaJiwvoYGzROfbOKb7Sd4YexQbovobnQkpyUXkzqI+Mt7UXymhr99vYsA\nb3f+3y2D5VZtYTVaa+au2M6neUeYdd0A4q/obXQkpybF7kAeHN2f4soaFv17P4HeJh685jKjIwkn\nMW/NHhb9ez9Jw3vz4Oj+RsdxelLsDkQpxf+7eQillbX89etdBPi4E395L6NjCQeX8Z+DvPqvXYyP\n7M7/u3mIvFO0AVLsDsbFRfHyHWGUVdcy5/Mt+HuZGBvezehYwkF9sfEocz7fwrWDO/GXO8JkqJeN\nkJOnDsjk6sJbk4cR07s9s5bl8/3Ok0ZHEg7o+50nmbUsn5je7Xlr8jBMrlIntkL+JhyUp8mVhQnR\nDOziy31Lcsk9eNroSMKB5B48zf1L8hjQ2ZeFCdF4mlyNjiR+Rordgfl5mkifHktXfy+SUrPZfqzM\n6EjCAew4XkZSajad/TxInx6Ln6cM9bI1UuwOrnGinre7G9MWZXHolAwNE6136FQl01Ky8HZ3IyM5\njiBfD6MjiQuQYncCPdp7k5EcS219A1NTMjlZVm10JGGHTpZXE78ok5r6BjKSY+nRXoZ62Sopdidx\nWWdf0pJiKao4y7RFWZRWytAw0XylVbVMS8misPwsqYkxXNbZ1+hI4ldIsTuRiB4BLIiPZm9hBdPT\ns6msqTM6krADVTX1JKdls7ewgnfiz31Eo7BtUuxO5srLOvL3uyPZcKiY+5fkUVMnQ8PExdXWN3D/\ne7nkHSrm73dHMuKyIKMjiWaQYndCN4V25cXxofywq5DHPtooQ8PEBTU0aB77aCPf7yzkxfGh3BTa\n1ehIopnkzlMnNSm2JyWVtby8agf+XiZeuG2o3Aoummitee7LrXyef5QnxgxkUmxPoyOJFpBid2L3\nX9WPksoa3lm7j0Afd2ZdN8DoSMJGvPbNbhb/dJCZI/ty/6h+RscRLSTF7uSeunEQxZU1vPHtbgK9\nTSQN72N0JGGw1H/v541vd3NndDBP3zhI3snZISl2J6eU4k/jQymtquX5L7fh72ViwrBgo2MJgyzf\nUMDzX27jhqGd+dP4UCl1O9Xmk6dKqR5KqTVKqW1Kqa1KqYfNEUxYj5urC3+/O5Lf9OvA4x9v4ptt\nJ4yOJAzw7fYTPPbRJn7TrwN/vzsSNxnqZbfM8TdXB/xeaz0EuBx4QCk1xAzrCivyNLmyYFo0Q7v5\n8cDSPDL3nTI6krCizH2n+O17eQzt5seCaTLUy961udi11se01nnnf14ObAfkww7tUDsPN9KSYgkO\n9GJGeg5bjpQaHUlYwdajpcxIzyE40Iu0pFjaecgRWntn1vdaSqneQCSQac51hfW093EnIzkOX083\nElOz2F90xuhIwoL2F50hYVEWvp7nhnq193E3OpIwA7MVu1KqHfAJ8IjW+hfzYZVSM5VSOUqpnMLC\nQnNtVlhAtwAvMmbEoTVMXZjJsdIqoyMJCzheWs3UhZk0aMiYEUe3AC+jIwkzMUuxK6VMnCv197TW\nn17oMVrrBVrraK11dFCQ3JZs6/oFtSMtKbZp+FPxmRqjIwkzKj5TQ3xKJqVVtaQnxdIvqJ3RkYQZ\nmeOqGAWkANu11n9reyRhK0KD/Xl3WjQHT1eSlJbNmbMyNMwRnDlbR1JaNgdPV/LutGhCg/2NjiTM\nzBx77MOBeGC0Uir//I+bzLCusAFX9OvAW5Mi2XyklHszcjlbV290JNEGZ+vquW9JLpsKSnhrUiRX\n9OtgdCRhAea4Kma91lpprcO01hHnf6w0RzhhG64f2oWXbw9j/Z4iHl2WT70MDbNL9Q2aWcs2sm53\nES/fHsb1Q7sYHUlYiFzXJJrljqhgSiprmLtiO/5em+WuRDujteYPn21hxeZj/OHmwUyM7mF0JGFB\nUuyi2WaM6EtxZQ3z1uwlwNudJ8cMMjqSaKZXVu/k/axDPHB1P2aM6Gt0HGFhUuyiRR67fiDFlbW8\n/f1eAr1NzBwpk/9s3btr9/GP7/cyOa4nj10/0Og4wgqk2EWLKKX4420hlFbV8qeVOwjwcufOGHlb\nb6s+zDnMiyu3c3NYV/54W4gcPnMSUuyixVxdFK/dGUFZVS1PfboJPy8TY0LkRJytWb31OE99sokR\nl3XktTsjcHWRUncWMr5NtIq7mwvvxEcR3iOAh97fwI97ioyOJH7mx71FPLh0A+E9AngnPgp3N/lf\n3ZnI37ZoNW93N1ITY+jT0Yd7Fuew8XCJ0ZEEsKmghHvSc+jd0ZvUxBi83eWNubORYhdtEuDtzuLk\nWAJ93ElMzWLPyXKjIzm1PScrSEzNJvD8MLcAbxnq5Yyk2EWbdfbzZElyHK4uLsSnZHGkRIaGGeFI\nSRXxKZm4KMWS5Dg6+3kaHUkYRIpdmEXvjj4snh5Lxdk64lMyOVVx1uhITuVUxVniUzKpOFvH4umx\n9O7oY3QkYSApdmE2Q7r5sSgxhqMlVSSkZlFeXWt0JKdQXl1LYmo2R4qrSEmIYUg3P6MjCYNJsQuz\niundnrenRLHjWDn3LM6hulaGhllSdW09Mxfnsv1YGW9PHUZsn/ZGRxI2QIpdmN3Vgzrx6sRw/rPv\nNA++v4G6+gajIzmkuvoGHnp/Az/tO8WrE8MZPaiz0ZGEjZBiFxYxLrI7z48dytfbTvDUp5tpkImQ\nZqW15ulPN/OvbSd47tYhjIuUjxkW/0cucBUWk/Cb3hRX1vD6N7sJ8DIx++bBcku7GWit+dPK7XyU\nW8DD11xG4vA+RkcSNkaKXVjUw9dcRkllLQvX7yfQx50Hru5vdCS794/v9/Luuv0kXNGLR669zOg4\nwgZJsQuLUkox55YhlFTW8MrqnQR4m5gS18voWHZraeYhXlm9k9siuvHsrUPlHZC4ICl2YXEuLopX\nJoZTVl3HHz7bgr+XiVvCuhkdy+6s2HSM2Z9t5uqBQbw6MRwXGeolLkJOngqrMLm6MG/yMKJ7BfLo\nsnx+2FVodCS7snZXIY8s20B0r0D+MSUKk6v8rysuTl4dwmq83F1ZmBBD/06+3JeRS96hYqMj2YW8\nQ8Xcm5FL/06+LEyIwcvd1ehIwsZJsQur8vcykT49hk5+HiSlZrPzuAwN+zW7TpSTlJpNJz8P0qfH\n4O9lMjqSsANS7MLqOvmeGxrmaXIhPiWTw6crjY5kkw6friQ+JRMPNxeWJMfRyVeGeonmkWIXhujR\n3pvF0+M4W9fA1JRMTpZXGx3JphSWnxvqVV3bQEZyHD3aexsdSdgRKXZhmIFdfElNiuFk2VkSFmVT\nWiVDwwBKq2qZtiiLE2VnWZQYw8AuvkZHEnbGLMWulFqklDqplNpijvWE8xjWM5B34qPYc7KcGenZ\nVNU499Cwqpp67knPYc/JcubHRxHVK9DoSMIOmWuPPQ0YY6a1hJMZOSCI1+6KIOdgMQ8szaPWSYeG\n1dY38MDSPLIPnuZvd0YwakCQ0ZGEnTJLsWut1wKnzbGWcE63hHXjxXGhfLfjJI99tNHphoY1NGge\n/2gj3+04ydxxIdwaLjdwidaTO0+FzZgc15PixtEDXiaeG2vALfPfvgBHN1h1kxrYd7KCCSVVzOrq\nQ89d3rDLqhGENY3+A3SPsugmrFbsSqmZwEyAnj17Wmuzws789qp+FJ+paRoa9si1A6wbIPMdcPeB\nAOu9Ro+XVlNWWkVfP0+6+9TDWbm236E1WP5Qo9WKXWu9AFgAEB0d7Vzvs0WzKaWYffNgSqpqm8b9\nWnUsrW6AsDvh+rlW2Vz6jwd49out3D4smFfuCEPJ/BdhBnIoRtgcpRQvTQilrKqW577cRoC3u/U+\nSKKhHpR1rgL+PP8Iz36xleuGdObl20NlqJcwG3Nd7vg+8BMwUClVoJRKNse6wnm5ubrwxqRILu/b\nnsc+2sh3O05YZ8O6AZTlZ7Gs2XGS33+4kbg+7XlzUiRuMtRLmJG5roqZpLXuqrU2aa2DtdYp5lhX\nODdPkyvvTotmcFc/7l+SR/YBK1x4pRssvseefeA097+Xy6CuvixMiMbTJEO9hHnJboKwab6eJtKS\nYuge6MX0tGy2HS2z7Aa1ZQ/FbDtaxvS0bLr5e5GWFIuvpwz1EuYnxS5sXod2HmQkx9HOw41pi7I4\nUHTGMhvS58/pu1hmD/pA0RmmLcqinYcbGTPi6NjOwyLbEUKKXdiF7gFeZCTHUt9wbmjYiTILDA3T\n5y9Ds8Ae+4myaqamZFLf0EBGcizdA7zMvg0hGkmxC7vRv5Mv6dNjKT5TQ3xKJiWVNebdQMP5OTVm\nLvaSyhqmpWRRfKaGtKRY+neSoV7CsqTYhV0JCw7g3WnRHCiqJCktm8qaOvMtboE99sqaOqanZbO/\n6AwLpkUT3iPAbGsLcTFS7MLu/KZ/R96YFMnGwyXcm5FLTZ2Z7uTT5t1jr6lr4L4leeQfLuGNSZEM\n79/RLOsKcSlS7MIujQnpwksTwli3u4hHP8yn3hxDwxr32M1w8rS+QTPrw3zW7irkpQlhjAnp0uY1\nhWguufNU2K07Y3pQUlXDn1buwN/LxIvjQto2NMxMh2K01sz5fAv/3HSMp28cxJ0xPdq0nhAtJcUu\n7NrMkf0orqzl7e/3Euht4vEbBrV+MTOdPP3rv3bxXuYh7hvVj3tH9WvTWkK0hhS7sHtP3DCQksoa\n5q3ZS6C3OzNG9G3dQo3XsbdhpMDCdft4a80eJsX24MkxA1u9jhBtIcUu7J5SirnjQimtqmXuiu0E\neLtzR1RwyxdqOnnausM5n+QWMHfFdm4M6cLccaHWnyUvxHlS7MIhuLooXrsrgvLqHJ78ZBN+nm5c\nP7SFJyzbcIz9620neOKTTVzZvyOv3x2Bq0xqFAaSq2KEw/Bwc2X+1ChCu/vzu/c38NPeUy1boJVX\nxfy09xQPLM0jpLs/78RH4eEmQ72EsaTYhUPx8XAjNTGGXu29uWdxDpsLSpv/za04ebrlSCn3LM6h\nZ3tv0hJj8PGQN8HCeFLswuEE+riTkRyHv5eJhNQs9hZWNO8bmw7FNG+Pe19hBQmLsvD3MpGRHEug\nj3srEwthXlLswiF18fdkyYw4XBTEL8zkaEnVpb+pBcfYj5ZUEZ+SBUBGcixd/WWol7AdUuzCYfXp\n6ENaUizl1XXEp2Ry+swlhoY1s9hPnx9CVlZVS/r0WPoGtTNTYiHMQ4pdOLSQ7v4sTIimoLiKxNQs\nKs7+ytCwZpw8rThbR1JqFgXFVSxMiCaku7+ZEwvRdlLswuHF9e3AvMnD2Hq0jJmLc6iurb/wAxt+\n/Tr2s3W41AzJAAAOxklEQVT13JuRw5ajZcybPIy4vh0slFiItpFiF07h2iGdeXViGD/uPcXDH2yg\nrv4CEyF/5VBMXX0DD7+fz7/3nOKVO8K4dkhnCycWovWk2IXTGB8ZzLO3DmH11hM8s3wzWv/PRMiL\nXBWjtWb28i2s2nqcObcMYcKwVtzVKoQVyUW3wqkkDe9DcWUtb3y7m0Bvd56+afD//eZF5rG/tGoH\ny3IO89Do/ky/so8V0wrROlLswuk8eu1llFTW8M7afQR4u3P/VecnMF7gUMz8H/byzg/7iL+8F49e\nN8CAtEK0nBS7cDpKKZ67dSgllbW8vGoHAd4mJsX2/MVVMR9kHeKlr3Zwa3g3nh87VIZ6CbthlmPs\nSqkxSqmdSqk9SqmnzLGmEJbk4qL4653hXDUwiNnLN7Ny8zFo+L899q82H+OZ5ZsZNSCIv04Mx0WG\negk70uZiV0q5AvOAG4EhwCSl1JC2riuEpZlcXXh7ShTDegbyyAf5bDx8GoAtxyp4+IN8InsGMn9q\nFO5uco2BsC/meMXGAnu01vu01jXAB8BtZlhXCIvzcnclJSGGvkE+vLpqGwCvfbObvkE+LEqIwctd\nJjUK+2OOYu8OHP7ZrwvOf00Iu+DvbWLx9FgCvM6dcvL18mDx9Fj8vU0GJxOidaz2HlMpNVMplaOU\nyiksLLTWZoVolk5+nvzhpnOfl/rUzUPp5OdpcCIhWs8cxX4E+PnHsAef/9p/0Vov0FpHa62jg4KC\nzLBZIcyrc7tze+xd/L0NTiJE25ij2LOBy5RSfZRS7sDdwBdmWFcI62rDR+MJYUvafB271rpOKfU7\nYDXgCizSWm9tczIhrK1xxEAzP2hDCFtllhuUtNYrgZXmWEsIw1xkpIAQ9kbuPBWiUdOhGGNvRqqt\nraWgoIDq6mpDcwjjeHp6EhwcjMnUuiuzpNiFaNQ4j/1XPmjDGgoKCvD19aV3794yxsAJaa05deoU\nBQUF9OnTuqFz8p5TiEY2cvK0urqaDh06SKk7KaUUHTp0aNM7Nil2IRpdZB67EaTUnVtb//6l2IVo\nJCdPm7Rr98sP6F67di3Dhg3Dzc2Njz/+uEXfK6xLXsFCNGq63FH+t7iQnj17kpaWxuTJk42OIi5B\nXsFCNLKRk6e2qnfv3oSFheHi0rza0Frz+OOPExISQmhoKMuWLQPg2LFjjBw5koiICEJCQli3bh31\n9fUkJiY2Pfa1116z5FNxeHJVjBCNbORyx597/sutbDtaZtY1h3Tz49lbh5p1zQv59NNPyc/PZ+PG\njRQVFRETE8PIkSNZunQpN9xwA7Nnz6a+vp7Kykry8/M5cuQIW7ZsAaCkpMTi+RyZ7LEL0chGropx\nFOvXr2fSpEm4urrSuXNnRo0aRXZ2NjExMaSmpvLcc8+xefNmfH196du3L/v27ePBBx9k1apV+Pn5\nGR3frskeuxCNmk6e2s6hGGvsWVvbyJEjWbt2LStWrCAxMZFZs2Yxbdo0Nm7cyOrVq5k/fz4ffvgh\nixYtMjqq3ZJdEyEayR67WY0YMYJly5ZRX19PYWEha9euJTY2loMHD9K5c2fuueceZsyYQV5eHkVF\nRTQ0NHD77bczd+5c8vLyjI5v12SPXYhGUuxNKisrCQ4Obvr1rFmzGDFiBOPHj6e4uJgvv/ySZ599\nlq1bLz7vb/z48fz000+Eh4ejlOIvf/kLXbp0IT09nVdeeQWTyUS7du1YvHgxR44cISkpiYbznzv7\n5z//2eLP0ZEp3XiJlxVFR0frnJwcq29XiF+V9S6sfAwe2w3tOhkWY/v27QwePNiw7QvbcKHXgVIq\nV2sdfanvlV0TIRrJdezCQcgrWIhGcuepcBDyChaikRxjFw5CXsFCNJJiFw5CXsFCNJKRAsJBSLEL\n0Uj22IWDkFewEI2k2JucOHGCyZMn07dvX6KiorjiiitYvnx5m9Z87rnnePXVVwGYM2cO33zzTavW\nyc/PZ+XKln/E8qRJkwgLC/vFgLGf5/q56dOn06lTJ0JCQi665sW+12jyChaikQ190IaRtNaMGzeO\nkSNHsm/fPnJzc/nggw8oKCj4xWPr6upatY0XXniBa6+9tlXf25piP378ONnZ2WzatIlHH320Wd+T\nmJjIqlWrWhPRcFLsQjSSPXYAvvvuO9zd3bnvvvuavtarVy8efPBBANLS0hg7diyjR4/mmmuuoaKi\ngmuuuYZhw4YRGhrK559/3vR9L774IgMGDODKK69k586dTV9PTExs+rCO3NxcRo0aRVRUFDfccAPH\njh0D4KqrruLJJ58kNjaWAQMGsG7dOmpqapgzZw7Lli0jIiKiaRRwo+rqapKSkggNDSUyMpI1a9YA\ncP3113PkyBEiIiJYt25ds/4cRo4cSfv27Zv955afn8/ll19OWFhY0x26AG+88QZDhgwhLCyMu+++\nG4AffviBiIgIIiIiiIyMpLy8vNnbaQ4ZKSBEo6aTpzZU7F89Bcc3m3fNLqFw40sX/e2tW7cybNiw\nX10iLy+PTZs20b59e+rq6li+fDl+fn4UFRVx+eWXM3bsWPLy8vjggw/Iz8+nrq6OYcOGERUV9V/r\n1NbW8uCDD/L5558TFBTEsmXLmD17dtMAsLq6OrKysli5ciXPP/8833zzDS+88AI5OTm89dZbv8g1\nb948lFJs3ryZHTt2cP3117Nr1y6++OILbrnlFvLz81vxB9Y806ZN480332TUqFHMmTOH559/ntdf\nf52XXnqJ/fv34+Hh0TSO+NVXX2XevHkMHz6ciooKPD09zZpFil2IRrrB6ffWL+SBBx5g/fr1uLu7\nk52dDcB1113XtDerteaZZ55h7dq1uLi4cOTIEU6cOMG6desYP3483t7eAIwdO/YXa+/cuZMtW7Zw\n3XXXAVBfX0/Xrl2bfn/ChAkAREVFceDAgUtmXb9+fdM7i0GDBtGrVy927dpl8THApaWllJSUMGrU\nKAASEhKYOHEiAGFhYUyZMoVx48Yxbtw4AIYPH86sWbOYMmUKEyZM+K+5PObQpmJXSk0EngMGA7Fa\naxkAI+yXLRb7r+xZW8rQoUP55JNPmn49b948ioqKiI7+vxElPj4+TT9/7733KCwsJDc3F5PJRO/e\nvamurm7WtrTWDB06lJ9++umCv+/h4QGAq6trq4/nG23FihWsXbuWL7/8khdffJHNmzfz1FNPcfPN\nN7Ny5UqGDx/O6tWrGTRokNm22dZX8RZgArDWDFmEMJaud/oTpwCjR4+murqat99+u+lrlZWVF318\naWkpnTp1wmQysWbNGg4ePAicO0b92WefUVVVRXl5OV9++eUvvnfgwIEUFhY2FXttbe2vTowE8PX1\nvegx6REjRvDee+8BsGvXLg4dOsTAgQN//Qmbgb+/P4GBgU3H7zMyMhg1ahQNDQ0cPnyYq6++mpdf\nfpnS0lIqKirYu3cvoaGhPPnkk8TExLBjxw6z5mlTsWutt2utd176kULYAVvcYzeAUorPPvuMH374\ngT59+hAbG0tCQgIvv/zyBR8/ZcoUcnJyCA0NZfHixU17nsOGDeOuu+4iPDycG2+8kZiYmF98r7u7\nOx9//DFPPvkk4eHhRERE8OOPP/5qvquvvppt27Zd8OTpb3/7WxoaGggNDeWuu+4iLS2taa//18yd\nO5fg4OCmH3Du8sgrrriCnTt3EhwcTEpKyq+ukZ6ezuOPP05YWBj5+fnMmTOH+vp6pk6d2nQy96GH\nHiIgIIDXX3+dkJAQwsLCMJlM3HjjjZfM2BJmGdurlPoeeKy5h2JaPbb3h1dgy8ct/z4hmqPiBNTV\nwOyjhsaQsb0C2ja295LH2JVS3wBdLvBbs7XWn1/g6xdbZyYwE6Bnz57N/bb/1q4TBFn+bZVwUkED\noWu40SmEaLNLFrvWunV3EfxynQXAAji3x96qRaISzv0QQghxUXJAUQghHEybil0pNV4pVQBcAaxQ\nSq02TywhnJsRH1kpbEdb//7bdB271no50LbJQEKI/+Lp6cmpU6fo0KEDSimj4wgr01pz6tSpNt2N\nKneeCmFjgoODKSgooLCw0OgowiCenp5tuhtVil0IG2MymejTp4/RMYQdk5OnQgjhYKTYhRDCwUix\nCyGEgzHLSIEWb1SpQuCg1Tfcdh2BIqNDWJk8Z8fnbM8X7Pc599JaB13qQYYUu71SSuU0Z06DI5Hn\n7Pic7fmC4z9nORQjhBAORopdCCEcjBR7yywwOoAB5Dk7Pmd7vuDgz1mOsQshhIORPXYhhHAwUuyt\noJT6vVJKK6U6Gp3F0pRSryildiilNimlliulAozOZClKqTFKqZ1KqT1KqaeMzmNpSqkeSqk1Sqlt\nSqmtSqmHjc5kLUopV6XUBqXUP43OYglS7C2klOoBXA8cMjqLlXwNhGitw4BdwNMG57EIpZQrMA+4\nERgCTFJKDTE2lcXVAb/XWg8BLgcecILn3OhhYLvRISxFir3lXgOeAJzi5ITW+l9a67rzv/wP0PqR\nc7YtFtijtd6nta4BPgBuMziTRWmtj2mt887/vJxzRdfd2FSWp5QKBm4GFhqdxVKk2FtAKXUbcERr\nvdHoLAaZDnxldAgL6Q4c/tmvC3CCkmuklOoNRAKZxiaxitc5t3PWYHQQS5Gxvf/j1z68G3iGc4dh\nHEpzPrBcKTWbc2/d37NmNmF5Sql2wCfAI1rrMqPzWJJS6hbgpNY6Vyl1ldF5LEWK/X9c7MO7lVKh\nQB9g4/lPtQkG8pRSsVrr41aMaHaX+sBypVQicAtwjXbc62OPAD1+9uvg819zaEopE+dK/T2t9adG\n57GC4cBYpdRNgCfgp5RaorWeanAus5Lr2FtJKXUAiNZa2+MgoWZTSo0B/gaM0lo77Ef6KKXcOHdy\n+BrOFXo2MFlrvdXQYBakzu2hpAOntdaPGJ3H2s7vsT+mtb7F6CzmJsfYxaW8BfgCXyul8pVS840O\nZAnnTxD/DljNuZOIHzpyqZ83HIgHRp//u80/vycr7JzssQshhIORPXYhhHAwUuxCCOFgpNiFEMLB\nSLELIYSDkWIXQggHI8UuhBAORopdCCEcjBS7EEI4mP8PvPJHWErKDegAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fccac3be080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = gluon.loss.L1Loss()\n",
    "\n",
    "# getting data ready\n",
    "output = nd.arange(-5,5,0.01)\n",
    "output.attach_grad() # we need the gradient\n",
    "thelabel = nd.zeros(shape=(output.size)) \n",
    "with ag.record():    # start recording\n",
    "    theloss = loss(output, thelabel)\n",
    "theloss.backward()   # and compute the gradient\n",
    "\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='L1 loss')\n",
    "plt.plot(output.asnumpy(), output.grad.asnumpy(), label='Gradient of L1 loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move on to other losses, let's quickly consider what happens if we want to minimize the L1 loss. Consider the toy example where we have a number of labels $y_i$ and we want to fit *all* of them to a single scalar, say $f$. In this case we need to solve the minimization problem:\n",
    "\n",
    "$$\\mathop{\\mathrm{minimize}}_f \\sum_i |y_i - f|$$\n",
    "\n",
    "As we saw above, the gradient is either -1 or 1. Hence, for the gradients to the left and to the right of $f$ to cancel out we need *the same number of $y_i$* on either side. This is the definition of the *median*. Hence, minimizing the L1 loss means that we are computing the median (at least for constant predictions). In general, the L1 loss is very robust against outliers, since the gradients can never get too large. \n",
    "\n",
    "## Regression - L2 loss\n",
    "\n",
    "Taking the squared distance between observation and estimate tends to be the default choice in many problems. Often for convenience we multiply this loss by a factor of $\\frac{1}{2}$ to ensure that the derivatives look pretty. Here's the loss:\n",
    "\n",
    "$$l(y,f) = \\frac{1}{2} (y-f)^2$$\n",
    "\n",
    "For vectorial $f$ and $y$ this is the squared Euclidean distance between points. The L2 loss has a few other nice properties. By a similar argument as before we can see that $\\sum_{i=1}^m \\frac{1}{2} (y_i - f)^2$ is minimized by choosing $f = \\frac{1}{m} \\sum_{i=1}^m y_i$, i.e. by choosing the mean. Let's see what it looks like in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX6wPHvSScNCCS00EsgHQhNRHqvgi4EVFB3sa0N\nf1asuLoq9r6sHamLNBVFUBRwAQmQkNBDaAkQUiCFFDLJ+f1xI7KSQMokdzLzfp7Hh5k7d+59B3nO\ne++557xHaa0RQgjheJzMDkAIIYQ5JAEIIYSDkgQghBAOShKAEEI4KEkAQgjhoCQBCCGEg5IEIIQQ\nDkoSgBBCOChJAEII4aBczA7gSho3bqzbtGljdhhCCFFn7NixI11r7V+RfW06AbRp04aYmBizwxBC\niDpDKXWsovtKF5AQQjgoSQBCCOGgJAEIIYSDkgQghBAOShKAEEI4qAonAKXUJ0qpM0qphEu2zVVK\n7VdK7VZKrVBKNSjnu0eVUvFKqVillAzrEUIIG1CZO4DPgBF/2rYOCNVahwMHgcev8P2BWutIrXVU\n5UIUQghREyqcALTWG4HMP237QWttKX27FQi0YmxVUmgpZt7Gw2w/mnn1nYUQwsbkXyjGUlxSK+ey\n5jOA24DvyvlMAz8opXYopWZe6SBKqZlKqRilVExaWlqlgygpgU82H+Uf3+5D1jsWQtQ1b/14iBFv\nbaKgqLjGz2WVBKCUmg1YgAXl7HKt1robMBK4Ryl1XXnH0lrP01pHaa2j/P0rNJv5f9Rzc+ahYZ2I\nO3GOb3afqvT3hRDCLCfP5fPpr0cIa1EfD1fnGj9ftROAUmoGMAaYpsu55NZap5T+eQZYAfSs7nmv\nZGK3QDo39eGVtfsptNR8FhVCCGt49YcDaOChYZ1q5XzVSgBKqRHAI8A4rXVeOft4KaV8fn8NDAMS\nytrXWpydFE+M6sKJzHzmb6lwWQwhhDBNQkoWK3alcFvftgQ29KyVc1ZmGOgiYAsQpJRKVkrdDrwL\n+ADrSod4fli6b3Ol1JrSrzYBNiul4oDfgG+11t9b9VeU4bpO/vTr2Jh3fkokK6+opk8nhBBVprXm\nxTX7aFDPlbsHtq+181a4GqjWOrqMzR+Xs+9JYFTp6yQgokrRVdMTo7ow6u1NvPdzIk+M6mJGCEII\ncVU/H0jjv4czeHZsML4errV2XrueCdylmS+TugXy2a9HOZFZZg+VEEKYylJcwotr9tGmkSdTe7Wu\n1XPbdQIA42GKk5PxcEUIIWzNf3Ykc+hMLo+N7IybS+02yXafAJrVr8ft17ZlVexJ4pOzzA5HCCEu\nOl9o4bUfDhLVuiHDQ5rW+vntPgEA3Nm/PY283HhxjUwOE0LYjnkbk0jPLeSJ0V1QStX6+R0iAfh4\nuHL/kI5sScpgw4EzZocjhBCkZhcwb2MSo8Ob0a1VQ1NicIgEABDdsxXtGnvxwrf7KKqlOhtCCFGe\nN9YdxFJSwqPDO5sWg8MkAFdnJ2aP7sLhtPN8uVUmhwkhzLP3ZDZLY05wc+82tGpUO5O+yuIwCQBg\nUOcA+nVszJvrD3H2/AWzwxFCOCCtNXO+2UP9eq7cP7ijqbE4VAJQSvHUmGByCy28sf6g2eEIIRzQ\n9wmn2ZqUyaxhQdT3rL1JX2VxqAQA0KmJD9N6tWLBtuMcTM0xOxwhhAMpKCrmhTX76NzUh+geLc0O\nx/ESAMCDQzrh7e7C89/slWGhQoha8/HmIySfzefpMcG4OJvf/JofgQkaernxwJCObDqUzo/7ZFio\nEKLmpWYX8N6GRIaHNOGaDo3NDgdw0AQAcFPv1rT39+KFNfu4YJFhoUKImvXy9/uxFGtmjwo2O5SL\nHDYBuDo78eSYYI6kn+fz/x41OxwhhB2LPXGO5TtTuL1fW1OHff6ZwyYAgIFBAQwI8uftHw+Rnlto\ndjhCCDuktea5r/fg7+POPQM7mB3O/3DoBADw5Ohg8ouKee0HGRYqhLC+VbEn2XX8HI8MD8LbvcJL\nsNQKh08AHQK8uaVPGxZvP87u5HNmhyOEsCPnCy289N1+wgPrM6lboNnhXMbhEwDAA0M70tjbnadW\n7aGkRIaFCiGs452fEjmdXcAzY4Nxcqr9ap9XIwkA8PVw5YlRnYk7cY6lMSfMDkcIYQcSz+Tw0aYk\nbuweSPfWfmaHU6ZKJQCl1CdKqTNKqYRLtvkppdYppQ6V/llmXVOl1PTSfQ4ppaZXN3BrmxDZgp5t\n/Hj5+/2cy5M6QUKIqtNa88zqPXi6OfPoSPOqfV5NZe8APgNG/GnbY8CPWuuOwI+l7/+HUsoPeAbo\nBfQEnikvUZhFKcWcCSFkF1iYu1aWjxRCVN238af4NTGDh4cH0djb3exwylWpBKC13ghk/mnzeODz\n0tefAxPK+OpwYJ3WOlNrfRZYx+WJxHSdm/oyvU8bFv52XJaPFEJUSW6hhee/2UtIc99aX+S9sqzx\nDKCJ1vpU6evTQJMy9mkBXNq5nly67TJKqZlKqRilVExaWpoVwqucB4Z2pJGXO0+tSpAHwkKISnvn\nx0OkZhfy/IRQnG3wwe+lrPoQWBuV1arVamqt52mto7TWUf7+/laKrOJ8PVyZPbozsSfO8Z8d8kBY\nCFFxh1Jz+HjzESZHtTRtmcfKsEYCSFVKNQMo/bOs6mopwKW1TwNLt9mk3x8Iv/SdPBAWQlSM1pqn\nVxkPfh8ZEWR2OBVijQSwGvh9VM90YFUZ+6wFhimlGpY+/B1Wus0mKaV4brw8EBZCVNzXu0+xJSmD\nh0d0ppENP/i9VGWHgS4CtgBBSqlkpdTtwEvAUKXUIWBI6XuUUlFKqY8AtNaZwPPA9tL/5pRus1ld\nmv3xQHjn8bNmhyOEsGHZBUX845u9hLbwZWrPVmaHU2HKlhdEiYqK0jExMaadP7fQwtDXf6F+PVe+\nvvdaXG1gAQchhO15amUCC7YdY8XdfYlo2cDUWJRSO7TWURXZV1q0K/B2d+G5cSHsP2082BFCiD/b\ncewsX247xi192pje+FeWJICrGBbSlGHBTXhz/UFOZOaZHY4QwoYUFZcwe0U8TX09+L/hdePB76Uk\nAVTAc+NDcFaKJ1cmyBrCQoiLPtp0hP2nc3huXIjNlXquCEkAFdCsfj3+b3gQvxxM45vdp67+BSGE\n3TuekcdbPx5kWHAThoU0NTucKpEEUEG39GlDeGB9nvt6L1n5RWaHI4Qwkdaa2SvjcXFy4rnxIWaH\nU2WSACrI2Unx4vVhZJ4v5OXv95sdjhDCRKvjTrLpUDr/N6wTzerXMzucKpMEUAmhLepzW9+2LNx2\nnJijNj2NQQhRQ87lXeD5b/YS0bIBN/dpY3Y41SIJoJIeHNqJFg3q8djyeAqKis0ORwhRy/65Zj9n\n84p48XrbL/Z2NZIAKsnL3YUXJ4aReCaXd346ZHY4QohatOlQGktiTvDXfm0JaV7f7HCqTRJAFfTv\n5M8N3QP58JckElJk3QAhHEFuoYXHvoqnnb8XDw7pZHY4ViEJoIqeGh2Mn5cbDy/bTVFxidnhCCFq\n2Cvf7+dkVj6vTArHw9XZ7HCsQhJAFdX3dOWFCaHsO5XNhz8fNjscIUQN2paUwRdbjjHjmjZEtbHN\nBd6rQhJANQwLacrYiOa8/dMhDqbmmB2OEKIG5F8o5pGvdtPKz5OH62C5hyuRBFBNz44NxsfDlYf/\nE4dFuoKEsDuv/XCAYxl5vDQpDE+3ulfu4UokAVRTI293nh0XQlxyFp/8KhVDhbAnO4+f5eNfjzCt\nVyuuad/Y7HCsThKAFYwNb8bQ4Ca89sNBktJyzQ5HCGEFBUXFPLJsN818PXhsZGezw6kRkgCsQCnF\nCxNCcXdx4v+kK0gIu/Dm+kMknsnln5PC8fFwNTucGiEJwEoCfD14fkIoO4+f418bk8wORwhRDduP\nZvKvjYeZ0qMl/Tv5mx1OjZEEYEXjIpozOrwZb64/yJ6TMkFMiLrofKGFh5bGEdiwHk+OCTY7nBpV\n7QSglApSSsVe8l+2UuqBP+0zQCmVdck+T1f3vLZIKcU/xofSwNONWUviKLRIrSAh6poX1uzjxNk8\nXrsxsk4u8lIZ1U4AWusDWutIrXUk0B3IA1aUseum3/fTWs+p7nltVUMvN16ZFM6B1BxeX3fQ7HCE\nEJWw4cAZFm47zsx+7ejZ1n4mfJXH2l1Ag4HDWutjVj5unTKwcwDRPVsxb2MS26VstBB1wtnzF3h0\n2W6Cmvjw4FD7qPVzNdZOAFOAReV81kcpFaeU+k4pVXeX0KmgJ0d3oWVDTx5aGsf5QovZ4QghruKp\nVQmczbvA65Mj7KbWz9VYLQEopdyAccB/yvh4J9Baax0BvAOsvMJxZiqlYpRSMWlpadYKr9Z5ubvw\n6o0RnDibxwtr9pkdjhDiClbHneSb3ad4YEgnuyjzXFHWvAMYCezUWqf++QOtdbbWOrf09RrAVSlV\n5rQ6rfU8rXWU1jrK379uD7/q2daPmf3asXDbcdbvveyvRQhhA1LO5fPkini6tmrAHde1MzucWmXN\nBBBNOd0/SqmmSilV+rpn6XkzrHhumzVrWCeCm/ny8LI4UrMLzA5HCHGJ4hLNg4tjKS7RvPGXSFyc\nHWtkvFV+rVLKCxgKLL9k251KqTtL394AJCil4oC3gSlaa22Nc9s6dxdn3o7uSkFRCQ8tjaOkxCF+\nthB1wnsbEvntaCbPTwilTWMvs8OpdVZJAFrr81rrRlrrrEu2fai1/rD09bta6xCtdYTWurfW+r/W\nOG9d0SHAm2fGBrM5MZ15m2SWsBC2YMexTN768RATIpszsVug2eGYwrHud0w0uUdLRoY25dW1B4g7\ncc7scIRwaFn5Rdy3KJbmDYwSLo5KEkAtUUrx0sRwAnzcuX/xLnJlaKgQptBaM3tFPKezC3hrSle7\nLfRWEZIAalF9T1fenNKV45l5PLNqj9nhCOGQlu1I5pvdp5g1tBPdWjU0OxxTSQKoZT3b+vH3gR34\namcyq2JTzA5HCIeSlJbLM6v30LudH3f2b292OKaTBGCC+wZ3JKp1Q55YHi8LyAhRSwqKirln4S7c\nXJx4Y3Ikzk7K7JBMJwnABC7OTrwd3RU3FyfuXrCTgiKpGipETXvu6z3sO5XNG3+JpFn9emaHYxMk\nAZikeYN6vDE5kv2nc+R5gBA1bPnOZBb9doK7B7RnYOcAs8OxGZIATDQgKIC/D+zAkpgTLNuRbHY4\nQtilQ6k5zF6RQM+2fsxykCqfFSUJwGQPDOlI73Z+PLkyngOnc8wORwi7knfBwl0LduLl7sw70V0d\nrtTD1cjfhslcnJ14e0pXvN1duXvBDikdLYSVaK15ckUCh9NyeWtKV5r4epgdks2RBGADAnw9eDs6\nkiPp53liRTwOUiZJiBq1ZPsJlu9K4YHBnejbocziww5PEoCNuKZ9Yx4c0olVsSeZv9WhF1QTotp2\nJ5/j6dV76NexMX8f1MHscGyWJAAbcs/ADgzuHMCcr/eyLckhqmULYXXpuYXcMX8H/t7uvDWlq4z3\nvwJJADbEyUnxxpRIWvl5cs/CnZzKyjc7JCHqlKLiEu5ZsJPM8xf4183d8fNyMzskmyYJwMb4ergy\n75bu5F8o5s4vZZKYEJXx4pp9bDuSyUuTwght4ThLO1aVJAAb1CHAh9f+EknciXM8vSpBHgoLUQHL\ndybz6a9Hua1vW67v6pj1/StLEoCNGhHalHsHdWBpTDJfbjtudjhC2LSElCweXx5P73Z+PD6qs9nh\n1BmSAGzYg0M6MTDIn+dW72H70UyzwxHCJmWUPvRt5OXGe1O74SqTvSpM/qZsmJOT4s0pXWnp58md\n83dwIjPP7JCEsCmFlmLu/HIH6bmF/OvmKBp5u5sdUp1itQSglDqqlIpXSsUqpWLK+Fwppd5WSiUq\npXYrpbpZ69z2rH49Vz6aHkVRcQm3f76dnIIis0MSwiZorXlieQLbj57l1RsjCAuUh76VZe07gIFa\n60itdVQZn40EOpb+NxP4wMrntlvt/b354KbuHE47z72LdmEpLjE7JCFM98Evh/lqZzIPDOnI2Ijm\nZodTJ9VmF9B44Att2Ao0UEo1q8Xz12l9OzTm+fGh/HwgjRfW7DM7HCFM9X3CaV75/gDjIppz/+CO\nZodTZ1kzAWjgB6XUDqXUzDI+bwGcuOR9cuk2UUFTe7Xitr5t+fTXo1IuQjishJQsHlwSS2TLBrxy\nQzhK2dFM33PH4ZdXYNU9tXI6Fyse61qtdYpSKgBYp5Tar7XeWNmDlCaPmQCtWrWyYnj2YfboLhzN\nOM+zq/fQppEn/Tr6mx2SELUmNbuAv34eQ0NPY8Kkh6uz2SFVX2EO7F0NcYvg6CZjW9vrwHIBXGp2\nJrPV7gC01imlf54BVgA9/7RLCtDykveBpdv+fJx5WusorXWUv780bn/m7KR4O7orHQO8uXvBTllD\nQDiMnIIibv10O9kFRXw8owcBPnW4vHNJCST9DMvvgFc7waq7ITsFBs6G+3fD9K9rvPEHKyUApZSX\nUsrn99fAMCDhT7utBm4pHQ3UG8jSWp+yxvkdjbe7Cx9Nj6KeqzMzPv1NagYJu3fBUsJdX+7kQGoO\n70/rRpdmvmaHVDXph2D9c/BmGHwxHg58B+F/gdt+gHt3Qv9HoGHrWgvHWl1ATYAVpX1xLsBCrfX3\nSqk7AbTWHwJrgFFAIpAH3GqlczukwIaefHprDyb/ayszPtnO0jv7UL+eq9lhCWF1Wmse+2o3mxPT\nmXtDOAOC6tiavvlnIeEriF0EKTGgnKD9YBg2B4JGgat5C9QrW64zExUVpWNiLptSIC7xa2I6Mz79\njW6tGvL5bT3to09UiEu88v1+3v/5MA8N7cS9dWXET3ERJP4IcQuNq/ziCxAQDBHRxhW/T9MaO7VS\nakc5Q/EvY82HwMIEfTs05tUbI7h/cSwPLY3jneiuOEn9c2EnvthylPd/Pkx0z1Z1Y2GX0/HGlX78\nUjifBp6NIOo2o+FvFgE2NmJJEoAdGB/ZgtTsAl5cs58AX3eeHhNsX0PjhEP6PuE0z6zew5AuATw/\nPsR2/03nnoHdSyFuMaTGg5MrBI2AiKnQYUitPMytKkkAduJv/dpxOquQT349QmNvd+4ZWAeuloQo\nx6+J6dy3aBcRgQ14J7obLrZW4K2oAA5+Z1ztJ64HXQzNu8GoVyF0Enj6mR1hhUgCsBNKKZ4c3YXM\n84XMXXsAb3cXpl/TxuywhKi0HcfO8rcvYmjb2IvPbu1BPTcbea6lNSTHGP36CV9BQRb4NINr7jW6\neALqXhlqSQB2xMlJMffGCM5fKOaZ1Xvwcnfhhu6yMIaoO/aezObWT3/D38ed+bf3pIGnDXSfZCUb\n3TtxiyAjEVzqQZcxRqPfbgA42UiCqgJJAHbG1dmJd6K78tfPY3hkWRxebs6MDJOSS8L2JaXlcssn\n2/Byd+HL23sR4GviRK8L5/+YnXtkI6ChdV/o+wAEjwePOjoP4U8kAdghD1dn5t3SnZs//o37Fu/i\nI3cX+neSWdXCdqWcy+emj7ahNcy/vRct/TxrP4iSEji22ejX37sKis5DwzYw4DEInwx+bWs/phom\nCcBOebq58MmMHkTP28od82P4/Nae9GrXyOywhLhManYBN320jZxCC4v+1psOAd61G0DGYeNKP24J\nZB0HNx8InQiRU6FVH5sbumlNMhHMzqXnFjL5X1s4lVXApzN6SBIQNiU1u4Ap87ZyJruAL27vSffW\ntTR6Jv8c7FlhNPwnthmzc9sNMIZudh4NbibcgVhJZSaCSQJwAGdyCpj6722knM3n01t70FuSgLAB\ntd74F1vg8E9Go7//WyguBP/Of8zO9bWPRWUkAYjLpOUUEv3vraSczeeTGT3o016SgDBPrTb+qXsg\ndiHE/wdyU6FeQwi70Wj4m3e1uy4eSQCiTGk5hUz991aSz+bz8Ywormnf2OyQhAOqlcb/fLrR4Mcu\nhNO7wckFOg6HyGjjTxuenVtdkgBEudJyCpn20VaOZ+bx8fQe9O0gSUDUnpPn8pn20baaafwthXBw\nrdHFc+gHKLEY9XcipkLYDeDlGP/WJQGIK0rPLWTav7dxJOM8703txtDgJmaHJBxAUlouN3/8G9n5\nRXx2Ww/rNP5aQ8pOo9FPWGaUXvZuavTpR0RDk+Dqn6OOkWqg4ooae7uzeGZvZny2nTu/3MHcG8KZ\n2E1mDIuas/dkNrd8YozzXzSzN6Et6lfvgNknS2fnLob0A+DiYYzeiZhqjOZxlqatIuRvyUE19HJj\nwV97MfOLGGYtjSM7v4gZfe1voosw345jmcz4dDve7i58+ddetPev4jj/C3mw/xujXz/pZ0BDy94w\n9i0IngD1GlgzbIcgCcCBebsbk8XuW7SLZ7/eS3aBhXsHdbDdsruiztl0KI2ZX+ygaX0P5t/ek8CG\nlRxfX1ICx7cYBdj2rIILOVC/FVz3MERMgUbtayZwByEJwMF5uDrz/rRuPPpVPK+vO8jZvAs8NTpY\nFpUR1bZyVwoPL4ujvb8382/vhb+Pe8W/nJlkzMyNWwTnjoGbt3GVHzHFqMnjZGPloesoSQACF2cn\n5t4QTv16rnzy6xFOZxXwxuRIWV5SVInWmvd/PszctQfo1daPeTdHUd+zAutVF2TBnpVGo398C6Cg\nXX8YONuovunmVeOxOxpJAAIwSkk/PTaYFg3r8Y9v95L67638+5YoGnlX4qpNODxLcQnPrN7Dgm3H\nGRfRnLk3huPucoULiZJiSNpgFGDb/w1YCqBRRxj8tFGArb4MTqhJ1U4ASqmWwBdAE0AD87TWb/1p\nnwHAKuBI6ablWus51T23sL7br21LiwYe3L84lokf/JfPbu1J28Zy5SWuLu+ChXsX7uLH/We4s397\nHhkeVH5X4pn9Rr/+7qWQcwo8GkDkNKMAW4vudjc711ZVex6AUqoZ0ExrvVMp5QPsACZorfdess8A\n4P+01mMqc2yZB2Ce31dl0loz75YoerSpG0vcCXOcyspn5hc72HMyi+fGh3Jz79aX73Q+w1hJK24h\nnNwFyhk6DjP69YNGgovcbVpDrc4D0FqfAk6Vvs5RSu0DWgB7r/hFYdO6t27I8ruu4dbPtjP131v5\nx4RQJvdoZXZYwgbtPH6WO+bvIK/Qwryboxhy6cRCywVjVm7cImOWbkkRNA2D4f806vF4yzoVZrLq\nMwClVBugK7CtjI/7KKXigJMYdwN7yjnGTGAmQKtW0uCYqU1jL1bcfQ33LtrFo1/Fs/dkNk+OCcbV\n1hboFqZZtiOZJ5bH07S+Bwv+2otOTXyM2bmnYo1+/YRlkJcBXgHQ6w5jdm7TULPDFqWsVgpCKeUN\n/AK8oLVe/qfPfIESrXWuUmoU8JbWuuPVjildQLbBUlzCy9/v59+bjtC7nR/vT+uOn5f9FtMSV2cp\nLuGl7/bz0eYjXNO+Ee9N7UbDkkzYvcRo+NP2gbMbBI0y+vXbD5bZubWk1msBKaVcgW+AtVrr1yuw\n/1EgSmudfqX9JAHYluU7k3lseTz+3u7Mu6U7Ic2rOZ1f1EkZuYU8sCSWTYfS+WvvZjzW9jAu8YuN\nWvu6BAJ7Gv36oRON0suiVtXqMwBlTBv9GNhXXuOvlGoKpGqttVKqJ+AEZFT33KJ2TewWSHt/b+6Y\nv4Pr3/8vz44NIbpnS5k57EC2H83k3gU7aZsfz48d42m/bx3EZoNvIFw7y+jiadzB7DBFBVnjnqwv\ncDMQr5SKLd32BNAKQGv9IXADcJdSygLkA1O0LZchFeWKaNmAb+67lgeXxPLEini2JmXw4sQwvN3l\n9t6eaa1Z9MNmUjd9xleum2nhchpOe0HwOKPRb9NPZufWQVIOWlRJSYnm/Z8TeX3dQdo08uK9ad3o\n0szX7LCEtRXmkBe7nBMbPiaoIA4AS6trcek2DbqMA/daXsBdXJWsByBqzZbDGdy3eBfZ+UU8OSaY\nm3q1ki6huq6kGI5shLhFFO9ZjXNxPkd0U852mETXMXegGpYxxl/YDEkAolal5RQya6nxUHBAkD+v\nTAonwNfD7LBEZaUd/GN2bnYKBc7efHWhF//1HsbM6MlEtJIHunWBJABR67TWzN96jBfX7MPD1ZkX\nrw9jVFgzs8MSV5OXWTo7dxGk7ADlTG7L/ryd0YPPM7pwQ68OzB7dBU83ecZTV0gCEKY5nJbLrCWx\nxCVncX3XFjw7NqRilSBF7SkugsT1xsIqB7+H4gsQEEJx+BS+ON+Tf246i4+7C6/cEM7gLrJcaF0j\nS0IK07T392bZXdfw7k+JvLshkU2H0nl2XDCjw5rJswGzndptXOnH/wfOp4FnY4i6HSKjSShuzSNf\nxbP3VAYjQ5syZ3xo5er3izpJ7gBEjUlIyeKx5btJSMlmcOcA5kwIpUWDemaH5VhyUiF+qbF2bmoC\nOLlC0Ahj7dyOQykoceKtHw8xb2MSfl5uPD8+hBGh0nVXl0kXkLAZluISPvvvUV774SBOCh4aFsTN\nfVpLPaGaVFQAB9YYV/uJP4IuNkosR0RD6CTwNCq7/rgvlTnf7OVYRh6To1ryxKgu0l1nByQBCJtz\nIjOPJ1cm8MvBNDoGePPM2BCu7djY7LDsh9aQvN3o19+z3Fhdy6c5REw2Gn7/oIu7Hkk/z5yv97Dh\nQBrt/b2YMz6Uvh3k/4W9kAQgbJLWmh/2pvLCt/s4npnH0OAmPDm6C60byYIzVXbuBOxebHTxZCSC\nSz3oMhYio6Ftf3D6YzWu3EIL729I5KNNR3BzceL+wR2Zfk0b3FzkbsyeSAIQNq2gqJiPNx/hvQ2J\nWIo1069pzV0DOkiF0YoqzIV9Xxtj9o9sArSxUHpENASPB4//nZFdaClm4bbjvPtTIhnnLzCxWwse\nG9FZ5mrYKUkAok5IzS5g7toDLN+ZjKebC3/t15bbr22Lj4f0Q1+mpASObjL69feuhqLz0LCN8TA3\nYrLx+rKvaFbHneTVHw6QfDafPu0a8ejIzkS2bFDr4YvaIwlA1CmHUnN4fd1Bvks4TUNPV+4a0J5p\nvVrjJQXmIOOw0a+/ewlknQB3XwiZYDT8rXqXuXZucYnm2/hTvL8hkf2ncwhu5sujIztzXcfGMhTX\nAUgCEHW6KXZpAAAZhklEQVTS7uRzzF17gE2H0qlfz5Xp17RhxjVtHK9rKP+c8SA3dhEk/wbKCdoN\nNBZW6TwaXMseSnvBUsLKXSl88MthjqSfp72/F/cN7sjY8OblL84u7I4kAFGn7Tx+lg9+Psy6vanU\nc3Vmco+W3H5tW1r6eZodWs0pthgLqsQthP1roLgQ/Dsb/frhk8G3/LH5WXlFLI05wae/HuFkVgEh\nzX35+8AODA9pKg2/A5IEIOzCodQcPvwliVWxKRRrzaCgAG7q05r+Hf3tp2E7nWD06+9eCufPQD0/\nY7H0yGhoFllmF8/v9p3K5ostR1mxK4WCohJ6tvHjroHtGdDJX7p6HJgkAGFXTp7LZ9Fvx1n02wnS\ncwtp6VePqT1bM7FbC5rUxZEsuWlGOYa4hXA6HpxcoNMI42q/4zBwKb/LK7fQwpr4UyyLSea3o5l4\nuDoxIbIFN/dpLUt0CkASgLBTFywlrN1zmvlbj/HbkUyUgr7tGzM+sjkjQpva9ughS6FReC12ESSu\ngxKLcYUfORVCbwCvRuV/tbiErUmZfLUzme8TTpNfVEzbxl5M6dGSyT1a0sDTwZ6RiCuSBCDsXlJa\nLitjT7JyVwrHM/Nwd3FiUOcAhnRpwsDOAbbx4FhrSNlpXOnHL4OCc+DdFML/YjT8AV3K/WqhpZhf\nE9NZm5DKun2pZJ6/gI+HC2MjmjOpWyDdWjWQbh5RJkkAwmFordl14hwrd6Wwds9pUrMLcVLQvXVD\nBnVuwjXtGxHS3BeX2qw9lJXyx+zc9IPg4gGdx5TOzh0AzpcPb9VaczQjj82J6Ww+lMaviRnkFlrw\ncXdhUJcAhoc0ZVDnADxcnS8/nxCXqPUEoJQaAbwFOAMfaa1f+tPn7sAXQHcgA5istT56teNKAhCV\nobUmISWbdftSWb83lb2nsgHwcXehR1s/erfzI7JlQ0Ka+1p/jsGF87DvG+NqP+kXQEOrPka/fsgE\n8Pjf/vmi4hIOnM5h14lzxB4/x9akDFLO5QPQokE9ruvkz/CQJvRp3wh3F2n0RcXVagJQSjkDB4Gh\nQDKwHYjWWu+9ZJ+7gXCt9Z1KqSnA9VrryVc7tiQAUR1ncgrYmpTJlsMZbEvKICn9PGAMrGnX2Ivw\nwAZ0auJD28ZetPP3opWfZ+WusEtK4Ph/jX79vSvhQi40aGU0+hFTwK8dxSWa09kFJJ7J5VBqDoln\ncjmYmsOek9kUWkoAaOztRlRrP67t2JhrOzSmdSNP6d4RVVbbCaAP8KzWenjp+8cBtNb/vGSftaX7\nbFFKuQCnAX99lZNLAhDWdCangISULOKTs4lPOUd8Shap2YUXP1cKAnzcCfDxIMDHHX8fdxp7u+Pp\n7oyXmwv13JzxdHPGK/cEzY+vJPD4KrzyUihy9uSw/xB2+Y1kv1so6XkWTmcVcOpcPqk5hRSX/PHP\nvKGnKx0DfAhtUZ+urRoQ2bIBgQ3rSYMvrKa2VwRrAZy45H0y0Ku8fbTWFqVUFtAISP/zwZRSM4GZ\nAK1atbJCeEIYAnw8GNTZg0Gd/1jmMKegiKPpeRzJOE9SWi4pZ/M5k1PIyawC4pKzyDhfiNbgQx6j\nnbcy0XkTPZ0OUKIVm0tC+ap4PD+URJF/3gOnY+DjYZSzaFa/Hr3bN6JZfQ+a1a9HhwBvOgZ408hb\nVtkStsPmiq1orecB88C4AzA5HGHnfDxcCQusT1hgGWPoS4rRh3+ieNdCnA+uQVkKKGzQgbSgx8gN\nmkigTwseUIrHXZ3x8XDB081ZruRFnWKNBJACtLzkfWDptrL2SS7tAqqP8TBYCNtzZl9pAbalqNzT\nuHg0gK43QcRU3Ft0w18p/M2OUQgrsEYC2A50VEq1xWjopwBT/7TPamA6sAW4Afjpav3/QtSq8xmQ\nsMxo+E/FGrNzOww1hm52GgEu0nUj7E+1E0Bpn/7fgbUYw0A/0VrvUUrNAWK01quBj4H5SqlEIBMj\nSQhhLssFOLTWGK9/cC2UFEHTcBjxkjE711uu84V9s8ozAK31GmDNn7Y9fcnrAuBGa5xLiGrRGk7u\nMgqwxS+D/EzwCoBedxizc5uEmB2hELXG5h4CC1Ejsk8Zi6rELYK0/eDsDp1HGQurtB9U5uxcIeyd\n/KsX9qsoH/Z/a/TrJ20AXQKBPWHMGxByPdRraHaEQphKEoCwL1rD8S3Glf6elVCYDfVbQr+HIHwK\nNO5gdoRC2AxJAMI+nD1qPMyNW2S8dvWC4PHGKJ7W14JTLRaDE6KOkAQg6q6CbNi7ymj0j/0KKGjb\nD/o/Bl3Ggru32REKYdMkAYi6paQYjvxiFGDb9zVY8sGvPQx60ujiadDy6scQQgCSAERdkXbAuNKP\nWwI5J43yypHRRuXNwB5XXDtXCFE2SQDCduVlQsJXxiiekztBOUOHITDiReg0Elzr4HrAQtgQSQDC\nthQXwaF1xsIqB743Zuc2CYVhL0DYjeDT5OrHEEJUiCQAYT6t4fRuo18//j+Qlw6ejaHn34wunmbh\nZkcohF2SBCDMk3PaaPBjF8GZPeDsBkEjjUa/wxBwdjU7QiHsmiQAUbuKCuDAt0ajf/hHY3ZuiygY\n/RqETARPP7MjFMJhSAIQNU9rOPGb0a+fsAIKs8C3BfR9wLja9+9kdoRCOCRJAKLmnDtuDNuMWwSZ\nh8HV05igFRENba8Dp0oswC6EsDpJAMK6CnNh32pj6ObRTca2Nv2g3yyjNIO7j7nxCSEukgQgqq+k\nBI5uLJ2duxqK8qBhWxg4G8InQ8PWZkcohCiDJABRdemJRr9+3BLITgZ3X2OsfuRUaNlLZucKYeMk\nAYjKyT8LCcuNfv3k7aCcjAVVhj4HnUeDaz2zIxRCVJAkAHF1xRZjyGbsQjjwHRQXgn8XGDoHwv4C\nvs3MjlAIUQXVSgBKqbnAWOACcBi4VWt9roz9jgI5QDFg0VpHVee8opacji+dnbsUzqeBZyOIurV0\ndm6EdPEIUcdV9w5gHfC41tqilHoZeBx4tJx9B2qt06t5PlHTcs/8MTs3NR6cXKHTcKNfv8NQcHEz\nO0IhhJVUKwForX+45O1W4IbqhSNMYSk0unbiFhmF2HQxNO8KI+dC6CTwamR2hEKIGmDNZwC3AUvK\n+UwDPyilNPAvrfU8K55XVIXWkLLD6NdP+AoKzoFPM7jm7xAxFQI6mx2hEKKGXTUBKKXWA03L+Gi2\n1npV6T6zAQuwoJzDXKu1TlFKBQDrlFL7tdYbyznfTGAmQKtWrSrwE0SlZCWXrp27GDIOgYvHH7Nz\n2w2Q2bl1SFFREcnJyRQUFJgdijCBh4cHgYGBuLpWvWii0lpXKwil1AzgDmCw1jqvAvs/C+RqrV+9\n2r5RUVE6JiamWvEJ4MJ5Y/nE2IVwZCOgodU1xopawRPAw9fsCEUVHDlyBB8fHxo1aoSSB/IORWtN\nRkYGOTk5tG3b9n8+U0rtqOhAm+qOAhoBPAL0L6/xV0p5AU5a65zS18OAOdU5r6iAkhJjofS4RcbC\n6RdyoUFr6P8oREwBv7ZXP4awaQUFBbRp00YafweklKJRo0akpaVV6zjVfQbwLuCO0a0DsFVrfadS\nqjnwkdZ6FNAEWFH6uQuwUGv9fTXPK8qTcdjo3tm92CjG5uYDIROMfv1WfcDJyewIhRVJ4++4rPH/\nvrqjgDqUs/0kMKr0dRIQUZ3ziKvIPwd7VhhX+ye2AQraD4RBT0HnMeDmaXaEwg55e3uTm5v7P9te\nf/11PvroI1xcXPD39+eTTz6hdevLa0GV9V1R+2QmcF1VbIGkDaWzc9eApQAaB8GQZ40CbL7NzY5Q\nOKCuXbsSExODp6cnH3zwAY888ghLlpQ3OFCYTfoD6prUvfDDk/BGMCy4wUgCXW+Gv/0E92yDax+U\nxl+YZuDAgXh6GnecvXv3Jjk5+Yr7a615+OGHCQ0NJSws7GKyOHXqFNdddx2RkZGEhoayadMmiouL\nmTFjxsV933jjjRr/PfZO7gDqgvPpEL/MqLx5Kg6cXKDjMGPoZqfh4OJudoTCZM99vYe9J7Oteszg\n5r48Mzakyt//+OOPGTly5BX3Wb58ObGxscTFxZGenk6PHj247rrrWLhwIcOHD2f27NkUFxeTl5dH\nbGwsKSkpJCQkAHDu3GVVZ0QlSQKwVZYLcGitUZLh0FoosUDTcBjxklFy2aux2REKUa4vv/ySmJgY\nfvnllyvut3nzZqKjo3F2dqZJkyb079+f7du306NHD2677TaKioqYMGECkZGRtGvXjqSkJO69915G\njx7NsGHDaunX2C9JALZEazi502j0E5YZpZe9m0Dvu4yr/SZVvxoT9q06V+rWtn79el544QV++eUX\n3N2rdnd63XXXsXHjRr799ltmzJjBrFmzuOWWW4iLi2Pt2rV8+OGHLF26lE8++cTK0TsWSQC2IPsk\n7F5iNPzpB8DZ3aitHzkV2g0EZ/nfJOqGXbt2cccdd/D9998TEBBw1f379evHv/71L6ZPn05mZiYb\nN25k7ty5HDt2jMDAQP72t79RWFjIzp07GTVqFG5ubkyaNImgoCBuuummWvhF9k1aFrNcyIP93xr9\n+kk/gy4xVtEa8yaEXA/1GpgdoRBXlJeXR2Bg4MX3s2bNYs2aNeTm5nLjjTcCRjmX1atXl3uM66+/\nni1bthAREYFSildeeYWmTZvy+eefM3fuXFxdXfH29uaLL74gJSWFW2+9lZKSEgD++c9/1uwPdADV\nLgVRk+yuFITWcHyLMXRzz0q4kAP1WxozcyOioVF7syMUdci+ffvo0qWL2WEIE5X1b6DWSkGICso8\nUlqAbRGcOwauXqWzc6OhdV+ZnSuEMIUkgJpSkA17Vxr9+sf/Cyhoex0MfMKovunmZXaEQggHJwnA\nmkqKjf78uEWw7xuw5EOjDkZJhvDJ0KCl2REKIcRFkgCsIe2A0a+/eynknASP+kap5YipEBgla+cK\nIWySJICqysssnZ27yBi7r5yh41AY8U/oNAJcPcyOUAghrkgSQGUUF8GhH4yr/YNroaQImoTB8BeN\n2bneVx/3LIQQtkKGn1yN1nAyFr57FF4LgsVTjZLLPWfCnZvhrs3Q5x5p/IXDSU1NZerUqbRr147u\n3bvTp08fVqxYUa1jPvvss7z6qrFY4NNPP8369eurdJzY2FjWrFlT6e9FR0cTHh5+WaG5S+P63YkT\nJxg4cCDBwcGEhITw1ltvlXnMsr5rK+QOoDw5p40+/bhFcGYvOLtB0EijX7/DYHCu+jqcQtR1Wmsm\nTJjA9OnTWbhwIQDHjh0rc9KXxWLBxaXyTc2cOVVfODA2NpaYmBhGjRpV4e+cPn2a7du3k5iYWKH9\nXVxceO211+jWrRs5OTl0796doUOHEhwcXNWwa53cAVyqKN/o1//yBni9C6x7yhiuOfp1eOgA/OUL\nCBohjb9weD/99BNubm7ceeedF7e1bt2ae++9F4DPPvuMcePGMWjQIAYPHkxubi6DBw+mW7duhIWF\nsWrVqovfe+GFF+jUqRPXXnstBw4cuLh9xowZLFu2DIAdO3bQv39/unfvzvDhwzl16hQAAwYM4NFH\nH6Vnz5506tSJTZs2ceHCBZ5++mmWLFlCZGTkZesRFBQUcOuttxIWFkbXrl3ZsGEDAMOGDSMlJYXI\nyEg2bdp01b+DZs2a0a1bNwB8fHzo0qULKSkpV/xObGwsvXv3Jjw8nOuvv56zZ88C8PbbbxMcHEx4\neDhTpkwB4JdffiEyMpLIyEi6du1KTk7OVWOqLLkD0Nro0vl9dm5hFvgGGnX1I6KhcUezIxTi6r57\nDE7HW/eYTcNg5EtlfrRnz56LjV95du7cye7du/Hz88NisbBixQp8fX1JT0+nd+/ejBs3jp07d7J4\n8WJiY2OxWCx069aN7t27/89xioqKuPfee1m1ahX+/v4sWbKE2bNnXywEZ7FY+O2331izZg3PPfcc\n69evZ86cOcTExPDuu+9eFtd7772HUor4+Hj279/PsGHDOHjwIKtXr2bMmDHExsZW+q/q6NGj7Nq1\ni169el1xv1tuuYV33nmH/v378/TTT/Pcc8/x5ptv8tJLL3HkyBHc3d0vlrl+9dVXee+99+jbty+5\nubl4eFh/YInjJoCzx4wCbHGLIDMJXD2hyzhj+Gab62R2rhCVcM8997B582bc3NzYvn07AEOHDsXP\nzw8wuoyeeOIJNm7ciJOTEykpKaSmprJp0yauv/76i4vIjBs37rJjHzhwgISEBIYOHQpAcXExzZo1\nu/j5xIkTAejevTtHjx69aqybN2++eKfSuXNnWrduzcGDB/H19a3Sb8/NzWXSpEm8+eabVzxGVlYW\n586do3///gBMnz79Ys2k8PBwpk2bxoQJE5gwYQIAffv2ZdasWUybNo2JEyf+T90la6lWAlBKPQv8\nDfh9afontNaXPXlRSo0A3gKcMRaLL/uyoqYV5sDe1Uajf7T0Fq9NP+j3fxA8Dtx9TAlLiGor50q9\npoSEhPDVV19dfP/ee++Rnp5OVNQfJWi8vP6Y7b5gwQLS0tLYsWMHrq6utGnThoKCggqdS2tNSEgI\nW7ZsKfPz30tOOzs7Y7FYqvJzqqyoqIhJkyZdbKSr6ttvv2Xjxo18/fXXvPDCC8THx/PYY48xevRo\n1qxZQ9++fVm7di2dO3e2YvTWeQbwhtY6svS/shp/Z+A9YCQQDEQrpWrvKUlJMRzeAMvvgFc7waq7\nITsFBj4J9++GGd9A12nS+AtRCYMGDaKgoIAPPvjg4ra8vLxy98/KyiIgIABXV1c2bNjAsWPHAKPu\n/8qVK8nPzycnJ4evv/76su8GBQWRlpZ2MQEUFRWxZ8+eK8bn4+NTbp95v379WLBgAQAHDx7k+PHj\nBAUFXfkHl0Frze23306XLl2YNWvWVfevX78+DRs2vPh8Yf78+fTv35+SkpKLI4pefvllsrKyyM3N\n5fDhw4SFhfHoo4/So0cP9u/fX+kYr6Y2uoB6Aola6yQApdRiYDywt0bPmn6odHbuEqPBd68P4X8x\nRvG07Cmzc4WoBqUUK1eu5MEHH+SVV17B398fLy8vXn755TL3nzZtGmPHjiUsLIyoqKiLV7LdunVj\n8uTJREREEBAQQI8ePS77rpubG8uWLeO+++4jKysLi8XCAw88QEhI+YvgDBw4kJdeeonIyEgef/xx\nJk+efPGzu+++m7vuuouwsDBcXFz47LPPKrRwzT/+8Q/efPPNi+8XL17M/PnzCQsLIzIyEoAXX3zx\niiOPPv/8c+68807y8vJo164dn376KcXFxdx0001kZWWhtea+++6jQYMGPPXUU2zYsAEnJydCQkKu\nurxmVVSrHHRpF9AMIBuIAR7SWp/90z43ACO01n8tfX8z0Etr/ferHb9K5aALc+GL8ZASA8oJ2g82\n+vWDRoFrvcodSwgbJuWgRY2Xg1ZKrQealvHRbOAD4HlAl/75GnBbRU58hfPNBGaCsZhEpbl7G3X1\ng8cbV/w+ZYUuhBDiqglAaz2kIgdSSv0b+KaMj1KAS8tgBpZuK+9884B5YNwBVOTcl5k4r0pfE0II\nR1Kth8BKqWaXvL0eSChjt+1AR6VUW6WUGzAFKH+NOCGEELWiug+BX1FKRWJ0AR0F7gBQSjXHGO45\nSmttUUr9HViLMQz0E631lR/hCyEqRGuNkgENDskay/lWKwForW8uZ/tJYNQl79cAla/MJIQol4eH\nBxkZGTRq1EiSgIPRWpORkVHt2cGOOxNYiDouMDCQ5ORk0tLSrr6zsDseHh7Vnh0sCUCIOsrV1ZW2\nbduaHYaow6TgjRBCOChJAEII4aAkAQghhIOqVimImqaUSgOOmR1HJTUG0s0OopbJb3YM8pvrhtZa\na/+K7GjTCaAuUkrFVLQOh72Q3+wY5DfbH+kCEkIIByUJQAghHJQkAOtzxEp08psdg/xmOyPPAIQQ\nwkHJHYAQQjgoSQA1SCn1kFJKK6Uamx1LTVNKzVVK7VdK7VZKrVBKNTA7ppqilBqhlDqglEpUSj1m\ndjw1TSnVUim1QSm1Vym1Ryl1v9kx1QallLNSapdSqqx1TuyCJIAaopRqCQwDjpsdSy1ZB4RqrcOB\ng8DjJsdTI5RSzsB7wEggGIhWSgWbG1WNs2As9xoM9AbucYDfDHA/sM/sIGqSJICa8wbwCMZaCXZP\na/2D1tpS+nYrxspv9qgnkKi1TtJaXwAWA+NNjqlGaa1Paa13lr7OwWgUW5gbVc1SSgUCo4GPzI6l\nJkkCqAFKqfFAitY6zuxYTHIb8J3ZQdSQFsCJS94nY+eN4aWUUm2ArsA2cyOpcW9iXMCVmB1ITZJy\n0FWklFoPlLXi/GzgCYzuH7typd+stV5Vus9sjC6DBbUZm6h5Silv4CvgAa11ttnx1BSl1BjgjNZ6\nh1JqgNnx1CRJAFWktR5S1nalVBjQFogrXaUpENiplOqptT5diyFaXXm/+XdKqRnAGGCwtt/xxSlA\ny0veB5Zus2tKKVeMxn+B1nq52fHUsL7AOKXUKMAD8FVKfam1vsnkuKxO5gHUMKXUUSBKa13XCkpV\nilJqBPA60F9rbbdLVCmlXDAecg/GaPi3A1PteZ1rZVzJfA5kaq0fMDue2lR6B/B/WusxZsdSE+QZ\ngLCWdwEfYJ1SKlYp9aHZAdWE0gfdfwfWYjwMXWrPjX+pvsDNwKDS/7expVfHoo6TOwAhhHBQcgcg\nhBAOShKAEEI4KEkAQgjhoCQBCCGEg5IEIIQQDkoSgBBCOChJAEII4aAkAQghhIP6f7jw3uJa29N5\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fccac3de5f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = gluon.loss.L2Loss()\n",
    "\n",
    "with ag.record():    # start recording\n",
    "    theloss = loss(output, thelabel)\n",
    "theloss.backward()   # and compute the gradient\n",
    "\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='L2 loss')\n",
    "plt.plot(output.asnumpy(), output.grad.asnumpy(), label='Gradient of L2 loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regression - Huber's Robust loss\n",
    "\n",
    "Huber's Robust Loss is a cross between the L1 and the L2 loss. It behaves like an L2 loss close to zero. Beyond that, for discrepancies larger than $\\rho$ it behaves like an L1 loss. The scaling is set up in such a way as to ensure that the derivative is continuous. \n",
    "\n",
    "$$l(y,f) = \\begin{cases}\n",
    "\\frac{1}{2 \\rho} (y-f)^2 & \\text{ for } |y-f| < \\rho \\\\\n",
    "|y-f| - \\frac{\\rho}{2} & \\text{ otherwise}\n",
    "\\end{cases}$$\n",
    "\n",
    "If we minimize the loss something interesting happens (again, we're in the toy scenario that we just estimate a scalar). The number of cases with $y_i < f-\\rho$ and with $y_i > f+\\rho$ are going to cancel out, since their gradients are all $-1$ and $1$ respectively. For all the $y_i$ closer to $f$, the gradients will balance out like in the L2 loss case. In other words, $f$ will be the mean for all points closer than $\\rho$. This is pretty much what a *trimmed mean* estimator does. It ensures that a few outliers (very large $|y_i|$) won't break the estimate. Let's check it out in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdclXX/x/HXly3LATgRwS2iICBquc1yW1bu7rorTXM0\nTK0sK7NhmS1Hd3fr7gZHao7UsjR3prJU3LhxIKIgQ/b398fx9tdwIJzDBed8no+Hj0dwrvE+lO8u\nrnNdn0tprRFCCGE97IwOIIQQwryk2IUQwspIsQshhJWRYhdCCCsjxS6EEFZGil0IIayMFLsQQlgZ\nKXYhhLAyUuxCCGFlHIzYqbe3t/b39zdi10IIUWHFxMRc1Fr73G45Q4rd39+f6OhoI3YthBAVllLq\nZHGWk1MxQghhZaTYhRDCykixCyGElZFiF0IIKyPFLoQQVsZsxa6UsldKxSmlVplrm0IIIe6cOY/Y\nnwEOmHF7QgghSsAsxa6U8gV6A1+YY3s3E3fqMvM2HrXkLoQQwmIOnLtSJvsx1xH7R8AkoOhmCyil\nRiqlopVS0SkpKSXayfK4M8z46SA/JZwrYUwhhDDGTwnn6fnxFtbstXx/lbrYlVJ9gAta65hbLae1\n/lxrHa61Dvfxue0dsTf0cu9mBPtWZuLiPRy/mFWibQghRFk7fjGLiYt3E+xbmW7Nqlt8f+Y4Yr8b\n6KeUOgEsBLoqpSLNsN2/cXawZ+7wMBzsFaMjY7iaV2iJ3QghhNlczStkdGQM9vaKOcNCcXawt/g+\nS13sWuuXtNa+Wmt/YDDwq9Z6eKmT3USdKpX4aHArDiVnMGX5XrTWltqVEEKUitaaV5YncCg5g48G\nheBb1bVM9lshr2Pv1NiHZ7o14vvYMyzYedroOEIIcUMLd51maWwS47o2onMTy5+C+R+zFrvWeqPW\nuo85t3kz47s2omNjH15fuY89SWllsUshhCi2vUnpvLZiHx0aefNMt0Zluu8KecQOYGen+GhQCD4e\nzoyOjOVyVp7RkYQQAoC07DxGR8Xg7e7Ex4NbYW+nynT/FbbYAaq5OTFnWCgXMnJ47rt4iorkfLsQ\nwlhFRZrnv9tN8pUc5gwLpZqbU5lnqNDFDhBStwpT+zZn46EUZm9INDqOEMLGzdt0lF8PXuDVPoG0\n8qtqSIYKX+wAw9v4cX9IbT5cd5gtR0p285MQQpTWtsSLfPDzIfoF1+aRtvUMy2EVxa6U4u0BLWhU\n3Z3xC+I4m3bV6EhCCBtzPj2H8QviqO/jzjsDWqBU2Z5X/yOrKHYAVycH5g0PI79Q83RULHkFN51u\nIIQQZpVXUMTTUTFczS/ks+GhuDkb8jjp66ym2AEa+Ljz3kMtiT+dxlur9xsdRwhhI9758QCxp9KY\n8WBLGlb3MDqOdRU7QK8WtXiyfQD/2X6SFfFnjI4jhLByq/ac5ettJ3jsLn/6Btc2Og5ghcUOMLln\nU1r7V+XFpXs5kpxhdBwhhJVKvJDJ5CV7CPWrwsu9mhkd5zqrLHZHeztmDw3FzdmeUZExZOYWGB1J\nCGFlsnILGB0Zg7OjPXOGheLkUH7qtPwkMbMani58OiSU4xezmLx0jwwLE0KYjdaal5ftJTElk08G\nt6JW5UpGR/oTqy12gHYNvJh4X1NW7znH19tOGB1HCGElIn8/yYr4szx/T2PaN/I2Os7fWHWxA4zq\nVJ97mtXg7TUHiDl5yeg4QogKLv50GtNW7adLEx/GdGlodJwbsvpiV0rxwcBgalepxNNRsVzMzDU6\nkhCigrqUlcfTkTFU93Dhw0Eh2JXxcK/isvpiB6hcyZF5w0NJy85n/II4CmVYmBDiDhUWaZ5dFM/F\nzDzmDQ+limvZD/cqLpsodoDmtSvz5v1B/HY0lVm/HDI6jhCigvn01yNsPpzCa/0Caelbxeg4t2Qz\nxQ4wMLwug8LrMmfDUdYfSDY6jhCigth0OIWP1x9hQGgdhkb4GR3ntmyq2AHe6N+c5rU9eW5RPKdS\ns42OI4Qo586kXeWZhXE0qeHBW/cbO9yruGyu2F0c7Zk3LAyA0VEx5OQXGpxICFFe5RYU8nRULAWF\nmrnDQqnkZG90pGKxuWIH8PNy5cNBIew7e4XXV+4zOo4Qopx6a/UBdp9OY+bDLanv4250nGKzyWIH\n6NasBmO6NGDhrtN8F33a6DhCiHJmRfwZvt1+khEdAugRVMvoOHfEZosd4PnuTbirgRevLk9g39l0\no+MIIcqJw8kZvLh0L639qzKpR1Oj49wxmy52ezvFJ0NaUcXVkdGRsaRfzTc6khDCYJm5BYyKjMHN\n2YHZQ0NxtK94NVnxEpuZt7szc4eFcjbtKhO+202R3LwkhM3SWjN56R5OXMzi0yGtqOHpYnSkErH5\nYgcIq1eNl3s1Y92BZP61+ZjRcYQQBvl62wlW7znHxPua0q6Bl9FxSkyK/Zp/3u1P75a1eH/tQbYf\nTTU6jhCijMWcvMTbaw5wT7MajOpU3+g4pSLFfo1SihkPtsTf241xC2JJvpJjdCQhRBm5mJnLmKg4\nalepxAcDgyvETUi3IsX+B+7ODnw2PIys3ELGzo8lv7DI6EhCCAsrLNI8szCOy9mm4V6VKzkaHanU\npNj/onEND959sAW7Tlxmxo8HjY4jhLCwD385zLbEVN7sH0Tz2pWNjmMWUuw30D+kDo+2q8cXW4+z\nZu85o+MIISzk14PJzN6QyMBwXwa2rmt0HLMpdbErpVyUUjuVUruVUvuUUm+YI5jRpvQOJKRuFSYt\n2cPRlEyj4wghzOz0pWyeW7SbwFqeTOsfZHQcszLHEXsu0FVrHQyEAD2UUm3NsF1DOTnYMXdYKI72\nitGRMWTnFRgdSQhhJjn5hYyOiqFIaz4bHoaLY8UY7lVcpS52bfK/Q1rHa3+s4i6f2lUq8cmQVhy5\nkMmUZQlobRVvSwib98YP+0k4c4VZA0Pw83I1Oo7ZmeUcu1LKXikVD1wAftFa7zDHdsuDDo18eO6e\nxiyLO0PkjlNGxxFClNLSmCQW7DzFqE4N6B5Yw+g4FmGWYtdaF2qtQwBfIEIp9bcTVkqpkUqpaKVU\ndEpKijl2W2bGdmlI5yY+vPnDfuJPpxkdRwhRQgfOXWHK8r20rV+NF+5tbHQcizHrVTFa6zRgA9Dj\nBq99rrUO11qH+/j4mHO3Fmdnp/hwYAg+Hs6MiYrlclae0ZGEEHfoSk4+oyNj8HRx5NMhoThUwOFe\nxWWOq2J8lFJVrv1zJaA7YHUXgFd1c2Le8FBSMnJ5ZlE8hTIsTIgKQ2vNpMV7OH35KrOHhuLj4Wx0\nJIsyx/+yagEblFJ7gF2YzrGvMsN2y52WvlV4rV8gmw+n8OmvR4yOI4Qopi+2HOenfed5qWdTIgKq\nGR3H4hxKuwGt9R6glRmyVAhDI/yIOXGZj9cfoZVfVTo1rlinlYSwNTuOpfLuTwfpGVSTJ9oHGB2n\nTFjvSSYLUUrx1gMtaFLDg2cWxpF0OdvoSEKIm7iQkcPYBXH4VXPlvYdaVvjhXsUlxV4ClZzsmTc8\njMJCzZioWHILCo2OJIT4i4LCIsbNjyMjJ595w0PxcKn4w72KS4q9hAK83Xj/4ZbsTkpn+qoDRscR\nQvzFzJ8Ps+P4Jd5+oAVNa3oaHadMSbGXQo+gWozsWJ///n6S5XFnjI4jhLjm533n+WzTUYa28WNA\nqK/RccqcFHspTbqvCRH+1Xjp+70cOp9hdBwhbN7J1CwmLN5NizqVmdon0Og4hpBiLyUHeztmD22F\nm7MDoyNjyMjJNzqSEDYrJ7+QUZGx2CnF3GGhVjfcq7ik2M2guqcLs4e24uSlbCYv3SPDwoQwyKvL\nEzhw7gofDQqhbjXrG+5VXFLsZtK2vheT7mvCmr3n+XLrcaPjCGFzFu06xeKYJMZ1bUiXptWNjmMo\nKXYzGtmxPvcG1uDdHw+y68Qlo+MIYTMSzqTz6op9tG/ozbP3WO9wr+KSYjcjpRQzBwbjW7USY6Ji\nScnINTqSEFYvPTufp6Ni8XJz4uPBIdjb2cZNSLcixW5mni6OzB0WRvrVfMYviKOgsMjoSEJYraIi\nzYTF8ZxNMw338nK37uFexSXFbgGBtT1564EWbD+Wyge/HDY6jhBW67PNR1l34AJTejcjrF5Vo+OU\nG1LsFvJQmC9DIvyYt/Eov+xPNjqOEFbnt6MXmbn2EL1b1uKxu/yNjlOuSLFb0Gt9Awmq48nz38Vz\nMjXL6DhCWI3z6TmMXxBHgLcbMx60neFexSXFbkEujvbMGxaGnVKMiowlJ1+GhQlRWvmFRYydH0t2\nXiGfDQ/D3bnU08etjhS7hdWt5sqHg4I5cO4KU1ckGB1HiApvxo8HiT55mXcGtKBRDQ+j45RLUuxl\noGvTGozr2pDvopNYtOuU0XGEqLDW7D3HF1uP82i7evQPqWN0nHJLir2MPHtPY9o39ObVFftIOJNu\ndBwhKpxjKZlMWrKHkLpVmNLbNod7FZcUexmxt1N8PDgELzcnRkfFkJ4tw8KEKK7svAJGR8biaK+Y\nMywUJweprluRn04Z8nJ3ZvbQUM6l5TBhcTxFRTIsTIjb0VozZVkChy9k8PHgVtSpUsnoSOWeFHsZ\nC6tXlVd6N2PdgQvM23TU6DhClHtRO06xLO4Mz3ZrTEd5eHyxSLEb4NG7/OkbXJsPfj7EtsSLRscR\notzak5TGtB/206mxD+O6NjQ6ToUhxW4ApRTvDmhBfR93xi+I43x6jtGRhCh3LmflMToyFh8PZz4a\nFIKdDPcqNil2g7g5O/DZ8FCu5hcyZn4s+TIsTIjrioo0z30XT0pGLnOHhVLVzcnoSBWKFLuBGlb3\nYMaDLYk5eZl31hw0Oo4Q5cbsDYlsPJTCq30DCa5bxeg4FY4Uu8H6Btfmsbv8+WrbcVbtOWt0HCEM\nt+VICh+uO8z9IbUZ3sbP6DgVkhR7OfByr2aE+lVh8pI9JF7INDqOEIY5m3aVZxbG06i6O28PaCHD\nvUpIir0ccHKwY86wUJwd7RkdGUNWboHRkYQoc3kFRTwdFUtufiHzhofh6iTDvUpKir2cqFW5Ep8M\nbkViSiYvfb8XreXmJWFb3l5zgPjTabz/cDANfNyNjlOhSbGXI+0beTOhe2NW7j7Lf38/aXQcIcrM\nyt1n+ea3Ezx+dwC9WtQyOk6FJ8VezjzduSFdm1bnzVX7iTt12eg4Qlhc4oUMXly6h/B6VXmpV1Oj\n41iFUhe7UqquUmqDUmq/UmqfUuoZcwSzVXZ2ilkDg6nh6cLTUbGkZuYaHUkIi8nKLWBUZCyuTvbM\nHhqKo70ca5qDOX6KBcAErXUg0BYYo5SSmZqlUMXViXnDwkjNzOPZRfEUyrAwYYW01rz4/V6OpWTy\nyeBW1KzsYnQkq1HqYtdan9Nax1775wzgACAT8EuphW9l3ujfnC1HLvLx+iNGxxHC7L7dfpIfdp9l\nwr1NuKuht9FxrIpZf+9RSvkDrYAdN3htpFIqWikVnZKSYs7dWq3BrevyUJgvn6w/woZDF4yOI4TZ\nxJ66zPTV++nWtDqjOzUwOo7VMVuxK6XcgaXAs1rrK399XWv9udY6XGsd7uMjozeLQynFm/2DaFrT\ng+cWxXP6UrbRkYQotdTMXMZExVKzsguzBspwL0swS7ErpRwxlXqU1vp7c2xTmFRysuez4WEUFmrG\nzI8lt6DQ6EhClFhhkebZRfGkZuUxb1gYlV0djY5klcxxVYwCvgQOaK1nlT6S+Ct/bzdmDgxmT1I6\n037Yb3QcIUrs4/VH2HLkItP6NSeoTmWj41gtcxyx3w08AnRVSsVf+9PLDNsVf3Bf85o81ak+UTtO\n8X1sktFxhLhjGw5d4JP1R3gozJdBresaHceqlXoYg9Z6KyAnycrAxHubEH8qjZeX7SWwtidNa3oa\nHUmIYkm6nM1zi+JpWtODN/sHyXAvC5O7ASoQB3s7Ph3aCg8XR0ZHxnIlJ9/oSELcVm5BIU9HxVJY\nqPlseBiVnOyNjmT1pNgrmOoeLswZGsqpS9lMWrxHhoWJcu/NVfvZk5TOzIHB+Hu7GR3HJkixV0AR\nAdV4sUdTftp3ni+2HDc6jhA3tSwuicjfT/FUx/rc17ym0XFshhR7BfVkhwB6NK/Juz8dZMexVKPj\nCPE3h85n8NL3e4kIqMbE+5oYHcemSLFXUEop3n+4JX7VXBm7II4LGTlGRxLiuoycfEZHxuDu7Mjs\nIa1wkOFeZUp+2hWYh4sj84aHkpGTz9j5cRQUFhkdSQi01kxasoeTl7KZM7QV1T1luFdZk2Kv4JrW\n9OTtB1qw8/gl3v/5kNFxhODLrcf5MeE8k+5rQpv6XkbHsUlS7FZgQKgvw9r48a9Nx1i777zRcYQN\niz5xiXd/PMi9gTUY2bG+0XFslhS7lZjaN5CWvpV54bvdnLiYZXQcYYNSMnIZMz8W36qVmDkwWG5C\nMpAUu5VwdrBnztBQ7O0VoyJjuJonw8JE2SkoLGL8gjjSsvOZOywMTxcZ7mUkKXYrUreaKx8OCuFQ\ncgavLE+Qm5dEmZn1y2G2H0tl+v1BBNaWURdGk2K3Ml2aVGdc10YsjU1i4a7TRscRNmDd/mTmbjzK\nkIi6PBwuw73KAyl2K/RMt0Z0aOTNayv2sTcp3eg4woqdSs3mue/iCarjyWt9mxsdR1wjxW6F7O0U\nHw9uhbe7E6OjYkjLzjM6krBCOfmFjI6KQQHzhoXh4ijDvcoLKXYrVc3NiTnDQkm+ksNzi+IpKpLz\n7cK8Xl+5j31nr/DhoBDqVnM1Oo74Ayl2K9bKryqv9glkw6EU5m5MNDqOsCLfRZ9m4a7TjOnSgG7N\nahgdR/yFFLuVe6RtPfqH1OaDXw6z9chFo+MIK7DvbDqvLk/grgZePN9dhnuVR1LsVk4pxTsDWtDQ\nx53xC+M4l37V6EiiAku/ms/TUbFUcXXkkyGtsLeTm5DKIyl2G+Dq5MC84WHk5pueZJNXIMPCxJ3T\nWvPC4t2cuXyVucNC8XZ3NjqSuAkpdhvRsLo77z0UTNypNN5ec8DoOKIC+tfmY/yyP5mXejUjrF41\no+OIW5BityG9W9bi8bsD+Oa3E6zcfdboOKIC+f1YKu/9dJDeLWrx+N3+RscRtyHFbmNe6tWUsHpV\neXHpHhIvZBgdR1QAF67kMHZ+HP7ebsx4qKUM96oApNhtjKO9HXOGhlLJ0Z6n/htDZm6B0ZFEOZZf\nWMTY+XFk5Rbw2fAw3J0djI4kikGK3QbVrOzCp0NacfxiFi8u3SPDwsRNvb/2EDtPXOKdAS1oXMPD\n6DiimKTYbdRdDb2ZcG8TVu05x39+O2F0HFEO/ZRwjs83H+ORtvW4v1Udo+OIOyDFbsNGd2rAPc2q\nM331AWJOXjY6jihHjl/MYuLiPQTXrcIrfZoZHUfcISl2G2Znp/jg4RBqVXFhTFQsqZm5RkcS5cDV\nvEJGR8bgYK+YOywUZwcZ7lXRSLHbuMqujswbFsal7DzGL4yjUIaF2TStNVOW7+VQcgYfDW5FnSqV\njI4kSkCKXRBUpzJv9m/OtsRUPlp32Og4wkALdp7m+9gzjO/aiE6NfYyOI0pIil0AMKi1HwPDffn0\n10R+PZhsdBxhgL1J6by+ch8dG/swvlsjo+OIUjBLsSulvlJKXVBKJZhje8IY0/oHEVjLk+cW7eb0\npWyj44gylJadx+ioGLzdnfhoUIgM96rgzHXE/g3Qw0zbEgZxcbRn3vBQirRmdFQMOfmFRkcSZaCo\nSPP8d7tJvpLD3OFhVHNzMjqSKCWzFLvWejNwyRzbEsaq5+XGrIEhJJy5whs/7DM6jigDczcm8uvB\nC0ztE0hI3SpGxxFmIPcHi7/pHliD0Z0bMG/jUUL9qsqT561VUSHHV3/A/dH/4hG3Ijy3OcI2o0PZ\ngAe/gICOFt1FmRW7UmokMBLAz8+vrHYrSmhC98bEn0rjleUJNK9dmcDankZHEuaUvJ+8758mIDmO\nOIeW1GjRCmUv59XLhKu3xXehzDUnRCnlD6zSWgfdbtnw8HAdHR1tlv0Ky0nJyKX3J1uo5GTPyrHt\nqVzJ0ehIorQK8mDrLPTmmVzRlZhe9E9GjZlIg+oyB6YiUErFaK3Db7ecXO4obsrHw5m5w0I5c/kq\nExfvlmFhFV1SDHzeCTa+Q4x7J7rkvEfPIWOk1K2QuS53XABsB5oopZKUUk+YY7vCeOH+1XixZ1N+\n3p/M55uPGR1HlEReNqydAl/eA1fTWBv8MQ9deJzHu4fTtWkNo9MJCzDLOXat9RBzbEeUT0+0DyD2\n1GVm/HSQ5rUr076R5c8RCjM5tgl+GA+XT0D442yvP57R/91Pz6CajOnS0Oh0wkLkVIy4LaUUMx5s\nSaPqHoyOiuFIsjx5qdy7mgYrx8G3/UDZwWOrOdrmTUYtPkKj6h7MfDhYnoRkxaTYRbF4uDjy5WPh\nODvY8/h/dnFRJkGWXwfXwNy2EBcJdz8Do38j1bs1//x6F472ii8eDcdNnoRk1aTYRbH5VnXli0fD\nScnIZeS30XJnanmTmQKL/wkLh4CrFzy5HrpPIwcnnvw2muQrOfz7H+HUreZqdFJhYVLs4o6E1K3C\nhwNDiD2VxguLd1MkY36NpzXsXgRzWsPBVdD1FRi5EeqEXhsXEE/86TQ+GhRCK7+qRqcVZUB+HxN3\nrGeLWkzu0ZQZPx2kVmUXXu7VTM7XGiXtNKx6DhJ/Ad8I6D8bfJpcf3nGTwdZs/c8L/dqSs8WtQwM\nKsqSFLsokVGd6nM+/Sr/3nKcam7OjO7cwOhItqWoCKK/hHWvgy6CHjMgYgTY/f/Tjv616Sj/2nyM\n4W39GNGhvnFZRZmTYhclopTitb7NuZydz4yfDlLF1ZEhETIqokxcTDRd8XLqN6jfBfp+DFXr/WmR\nhTtP8c6PB+nTshZv9AuS36hsjBS7KDE7O8XMh4O5kpPPlGV7qerqSI8g+XXfYgoLYPunsOEdcHSB\n/nMhZCj8pbRX7znHS8v20qmxD7MGymx1WyQfnopScXKwY+6wUELqVmH8gng2HrpgdCTrdG4PfNHV\ndOql8b0wZhe0Gva3Ut98OIVnF8UR5leVz4aH4eQgf8VtkfxbF6Xm6uTA149F0KiGOyP/G8OmwylG\nR7Ie+Tmwfhp83hmunIOB38KgSPD4+yiArUcuMuLbaBpW9+DLx1pTycn+79sTNkGKXZhFZVdHop5s\nQ0Mfd0Z8Gy3lbg6nfofP2sOWDyB4CIzZAYH9b7jo5sMpPPGfXQR4uxH5RIRM4rRxUuzCbKq4Ov2p\n3DdLuZdMbgasmQhf9YCCXBj+Pdw/B1yr3XDxTYdTePLbaOr7uDN/RFu83J3LOLAob6TYhVlVdTOV\newMfd578Npp1+5ONjlSxJK6Due1g57+hzVPw9HZo2O2mi68/kGw6/eLjzvwn28jzSgUgxS4soKqb\nE/OfbEOzmh48FRnD0pgkoyOVf9mXYNkoiHwQHCvB42uh5wxwdr/pKktjkhj53xia1vRg/og2VJVS\nF9dIsQuLqOrmRNSItrStX40Ji3fzxRaZ5X5DWsO+5TAnAvYuho4T4akt4Nfmlqv9e/MxJizeTdv6\n1Zg/oi1VXKXUxf+T69iFxbg7O/DVY615dmE801cfIDUrj4n3NsFOrqs2yTgPqyeY5rvUCoFHlkHN\nFrdcpahI897aQ3y26Si9WtTkw0EhODvI1S/iz6TYhUU5O9gze2gor65IYN7Go5xMzeKDh0Ns+1I8\nrU0jdX+eYvpwtPs0aDsG7G/91/FqXiEvLNnN6j3nGNbGj2n9g+TmI3FDUuzC4uztFG/dH0SAlxtv\n/3iApMvb+eIf4VT3dDE6Wtm7dBxWPQvHNkK9u6Hfp+B1+zk7yVdyGPFtNHvPpPNSz6aM7FhfxgSI\nm5Jz7KJMKKUY0bE+nz8STuKFTPrN3sbepHSjY5WdokLYPhfm3WV6qHTvWfDoqmKVesKZdPrP3kbi\nhUw+fyScpzo1kFIXtyTFLspU98AaLBl1F3YKHpz3G1E7TqK1lc90v3AQvroP1r4E/h1gzO/Q+gmw\nu/VfP601C3aeYsC837C3UywdfRfdA+Xh0+L2pNhFmQus7cmq8R1o28CLKcsSeP673WTnFRgdy/wK\n8mDTe6a7R1OPwoAvYOgiqOx721Wzcgt4/rvdvPT9XtoEVGPl2LtpVsuzDEILayDn2IUhqrk58c1j\nrZm9IZEP1x0m4Uw6Hw9uRWBtKymvMzGwYhxc2AdBD5muSXfzLtaqB89fYez8OI6mZPJ898aM6dJQ\nPiQVd0SO2IVh7OwU47s14tvHI7icnU//OVuZsyGRgsIio6OVXF42/PwKfHEPXL0MQxbCQ18Wq9QL\nizRzNybS99OtpGXnE/lEG8Z3aySlLu6YMuL8Znh4uI6Oji7z/Yry61JWHq8s38uavecJ9avCBwND\nCPB2MzrWnTm+xfQAjMvHIeyf0P0NcKlcrFWPpWQyYfFu4k6l0atFTabf30LGA4i/UUrFaK3Db7uc\nFLsoL7TWrNx9lleXJ5BbUMSYLg15qlP98n8DTk46/DIVYr6BqgHQ7xMI6FisVXMLCvn35mPM3pCI\ns4M90/o3p19wbbnqRdxQcYtdzrGLckMpRf+QOrQJ8OLN1fuZ9cthlsWdYVr/5nRo5GN0vBs79COs\neh4yz8Nd46Dzy+DkWqxVtx65yNQVCRy7mEWvFjV5rW9zatjitf3C7OSIXZRbmw+nMHVFAidSs7mn\nWQ0m9WhC4xoeRscyyboIP06GhCVQvTn0/xTqhBVr1cPJGbz30yHWHUjG38uVN/oH0alxOf0flyhX\n5FSMsAo5+YV8ufU4n208SlZeAQNCfXn2nkb4Vi3eUbHZaQ17l8CPk0xz0ztNgrufBYfbnw9PupzN\nR+uO8H1sEm5ODozq3IAn2gfg4ljOTzWJckOKXViVy1l5zN2YyH+2n6SoSNMvpDZPdWxAk5pleASf\nnmQ67XJkLfi2No0DqN7stqsdOp/BvzYfZWX8WezsFI+2q8fTnRvKmF1xx6TYhVU6m3aVL7YcZ+Gu\nU2TnFdKliQ/D29ajU2MfHOwtdPVuURHEfA2/vAa6ELpNhYiRYHfzI+2CwiI2HU4h8veTbDiUQiVH\newZH1GW5pJiwAAAQiElEQVREh/rUrlLJMjmF1ZNiF1btclYe//39JN9uP8nFzFxqerrwcLgv/UPq\n0LD6zR9OccdSj5ouYTy5Dep3hr4fQ1X/my6eeCGTFfFnWBydxPkrOXi7O/OPdvV4pG09OUIXpVam\nxa6U6gF8DNgDX2it373V8lLswlzyC4tYf+ACi3adYtPhFIo0NKzuTs+gmnRrVoOg2p4lO5IvLIDt\ns2HjO2DvDPe9Ba2Gw18uQywoLCLh7BXWH0jmx4TzJF7IxE5Bp8Y+DGrtR7dm1XG01G8SwuaUWbEr\npeyBw0B3IAnYBQzRWu+/2TpS7MISzqfnsHbfeX5KOM+O46kUadPDPlr7VyUiwIvmtT1pWssDH3fn\nW18nfn4vrBgL5+KhaR/oNRM8a6G1JiUzl4PnMth39go7j6ey68RlMnMLsFPQJsCLHkE1ua95TWpW\nlssWhfmVZbG3A17XWt937euXALTW79xsHSl2YWmpmbn8djSV34+Z/hxNybr+mpebE3WruVKrsgs1\nPF3wdnfCxdEeV/tCQo7/m6aJX5Lr4MnGhpPY4dKB5IxczqXncPpSNqlZede308DHjTb1vWhb34u7\nG3jh5e5sxFsVNqQsb1CqA5z+w9dJwN8e2KiUGgmMBPDz8zPDboW4OS93Z/oG16ZvcG3ANLLg4Pkr\nHDqfwcFzGZxJu8rh5Aw2H04hK6+QUHWYGY7/ppHdGZYUdmT61WGkxXng5pREzcou1KpciXua1aBJ\nTQ+a1vKgaU1PueVflFtlduep1vpz4HMwHbGX1X6FANM0ybsaeHNXg78M48rNpGDdG9jv+jdFHnVI\n6Tqfjg26scnRHhdHu/I/zkCIGzBHsZ8B6v7ha99r3xOifEtcDz88i0P6aYgYgX23qfg4l5M7W4Uo\nBXMU+y6gkVIqAFOhDwaGmmG7QlhG9iXTaN34KPBqBI//BH5tjU4lhNmUuti11gVKqbHAWkyXO36l\ntd5X6mRCWML+FbD6BchOhQ4ToOMkcJQrWIR1Mcs5dq31GmCNObYlhEVkJMOaF+DASqjZEoYvhVot\njU4lhEXI2F5h3bSG+PmmB0nn58A9r0O7cWAv/+kL6yX/dQvrdfkk/PAMHNsAfneZHoDh3cjoVEJY\nnBS7sD5FhbDz37B+mmkEQK+ZEP4E2Mmt/cI2SLEL65JyyDQOIGknNOwOfT6EKnVvv54QVkSKXViH\nwnzY+hFsfg+c3OGBz6HlwL8N7RLCFkixi4rvbJzpKD05AZoPgJ7vgbs8ak7YLil2UXHlXzWN1f3t\nU3CvAYPnQ9PeRqeyWvn5+SQlJZGTk2N0FKvn4uKCr68vjo6OJVpfil1UTCe2mh6AcekYhD4K3adB\npSpGp7JqSUlJeHh44O/vf+uxx6JUtNakpqaSlJREQEBAibYhxS4qlpwrsO41iP7K9CSjf6yE+p2M\nTmUTcnJypNTLgFIKLy8vUlJSSrwNKXZRcRxeC6ueg4xz0G4sdJkCTq5Gp7IpUuplo7Q/Z7mwV5R/\nWRdh6ZMwfyA4e8ITv5geVSelbnPc3f/8PNtvvvmGsWPH3nKdjRs30qdPH7NleP3115k5c6bZtmcJ\ncsQuyi+tIWEp/DjJdAqm80vQ/nlwkAdciLJTWFiIvX3FmssvR+yifEo/AwsGw9InTOfSn9oMnV+U\nUhc39dhjj7FkyZLrX//x6P7KlSv07t2bJk2aMGrUKIqKigD4+eefadeuHaGhoTz88MNkZmYC4O/v\nz+TJkwkNDWXx4sXF2v+sWbMICgoiKCiIjz76CICsrCx69+5NcHAwQUFBLFq0CIAXX3yRwMBAWrZs\nyQsvvGCW9/9HcsQuypeiIoj9D/wy1XTT0X1vQ5tRYFexjpis3Rs/7GP/2Stm3WZgbU9e69v8lstc\nvXqVkJCQ619funSJfv363XbbO3fuZP/+/dSrV48ePXrw/fff07lzZ6ZPn866detwc3NjxowZzJo1\ni6lTpwLg5eVFbGxssbLHxMTw9ddfs2PHDrTWtGnThk6dOnHs2DFq167N6tWrAUhPTyc1NZVly5Zx\n8OBBlFKkpaUVax93QopdlB+pR01Du05sgYCO0PcTqFayy72EdapUqRLx8fHXv/7mm2+Ijo6+7XoR\nERHUr18fgCFDhrB161ZcXFzYv38/d999NwB5eXm0a9fu+jqDBg0qdq6tW7fywAMP4ObmBsCAAQPY\nsmULPXr0YMKECUyePJk+ffrQoUMHCgoKcHFx4YknnqBPnz5mPf//P1LswniFBfD7XNjwFtg7Q79P\nodUjMg6gHLvdkbURHBwcrp9iKSoqIi8v7/prf73KRCmF1pru3buzYMGCG27vfyVdGo0bNyY2NpY1\na9bwyiuv0K1bN6ZOncrOnTtZv349S5YsYfbs2fz666+l3tcfyTl2YazzCfDlPfDLq9CgG4zZAaH/\nkFIXd8zf35+YmBgAVq5cSX5+/vXXdu7cyfHjxykqKmLRokW0b9+etm3bsm3bNhITEwHT+fDDhw+X\naN8dOnRg+fLlZGdnk5WVxbJly+jQoQNnz57F1dWV4cOHM3HiRGJjY8nMzCQ9PZ1evXrx4Ycfsnv3\n7tK/+b+QI3ZhjIJc2DwTts4Clyrw0NfQ/AEpdFFiI0aMoH///gQHB9OjR48/HXG3bt2asWPHkpiY\nSJcuXXjggQews7Pjm2++YciQIeTm5gIwffp0GjdufNt9TZ8+/foHpGC6K/exxx4jIiICgCeffJJW\nrVqxdu1aJk6ciJ2dHY6OjsybN4+MjAz69+9PTk4OWmtmzZpl5p8EKK212Td6O+Hh4bo458WElTq9\nC1aOhZSD0HIw9HgHXKsZnUrcxoEDB2jWrJnRMWzGjX7eSqkYrXX47daVI3ZRdvKy4Nfp8Ps88KwD\nw5ZAo+5GpxLC6kixi7JxdAP8MB7STkHrEXDPa+DsYXQqIaySFLuwrKuX4edXIC4SvBrCP3+EencZ\nnUoIqybFLiznwA+weoJp1kv756HTZHB0MTqVEFZPil2YX0Yy/DgR9q+Ami1g6HdQO+T26wkhzEKK\nXZiP1rB7Afz0kunpRt2mwl3jwb5kT4ERQpSM3KAkzCPtFEQ+CMtHg09TGLUVOkyQUhdml5yczNCh\nQ6lfvz5hYWG0a9eOZcuWlXh7fxzDO3XqVNatW1ei7cTHx7NmzZoS5zAnOWIXpVNUBLu+gHWvm24u\n6jUTwp8AOzlmEOanteb+++/n0UcfZf78+QCcPHmSlStX/mm5goICHBzuvN6mTZtW4mzx8fFER0fT\nq1evEm/DXORvnyi5lMPwdU/T+XS/tvD0dogYIaUuLObXX3/FycmJUaNGXf9evXr1GDduHN988w39\n+vWja9eudOvWjczMTLp160ZoaCgtWrRgxYoV19d56623aNy4Me3bt+fQoUPXv//H0b8xMTF06tSJ\nsLAw7rvvPs6dOwdA586dmTx5MhERETRu3JgtW7aQl5fH1KlTWbRoESEhIdfH8xpFjtjFnSvMh20f\nw6YZ4OQG938GwYNlHIAt+fFFOL/XvNus2QJ6vnvLRfbt20doaOhNX4+NjWXPnj1Uq1aNgoICli1b\nhqenJxcvXqRt27b069eP2NhYFi5cSHx8PAUFBYSGhhIWFvan7eTn5zNu3DhWrFiBj48PixYtYsqU\nKXz11VeA6TeCnTt3smbNGt544w3WrVvHtGnTiI6OZvbs2aX/WZSSFLu4M2fjYcVYSN4LgfdDr/fB\nvbrRqYSNGjNmDFu3bsXJyYkxY8bQvXt3qlUzjafQWvPyyy+zefNm7OzsOHPmDMnJyWzZsoUHHngA\nV1fToxVvNM/90KFDJCQk0L276c7owsJCatWqdf31AQMGABAWFsaJEycs/C7vXKmKXSn1MPA60AyI\n0FrLABhrlX8VNr4Lv30Kbt4wKBKa9TU6lTDKbY6sLaV58+YsXbr0+tdz5szh4sWLhIebxqf8cfBX\nVFQUKSkpxMTE4OjoiL+/Pzk5OcXaj9aa5s2bs3379hu+7uzsDIC9vT0FBQUlfTsWU9qToQnAAGCz\nGbKI8urkb/BZe9j2EYQMNY3WlVIXBujatSs5OTnMmzfv+veys7NvuGx6ejrVq1fH0dGRDRs2cPLk\nSQA6duzI8uXLuXr1KhkZGfzwww9/W7dJkyakpKRcL/b8/Hz27dt3y2weHh5kZGSU9K2ZVamKXWt9\nQGt96PZLigop54rpztGve5rOq/9jBfSfDZWqGp1M2CilFMuXL2fTpk0EBAQQERHBo48+yowZM/62\n7LBhw4iOjqZFixZ8++23NG3aFIDQ0FAGDRpEcHAwPXv2pHXr1n9b18nJiSVLljB58mSCg4MJCQnh\nt99+u2W2Ll26sH///nLx4alZxvYqpTYCLxT3VEyJx/Zueh8Sltx+OWEemRdMs17aPg1dp5g+KBU2\nS8b2li2Lju1VSq0Dat7gpSla6xU3+P7NtjMSGAng5+dX3NX+zL06+DQp2briztVsCREjoe7fj2iE\nEOXXbYtda32POXaktf4c+BxMR+wl2kjYo6Y/QgghbkruJBFCCCtTqmJXSj2glEoC2gGrlVJrzRNL\nCFEeGfEoTVtU2p9zqa5j11ovA0o+fUcIUWG4uLiQmpqKl5cXSu4ythitNampqbi4lPzZBXLnqRCi\nWHx9fUlKSiIlJcXoKFbPxcUFX1/fEq8vxS6EKBZHR0cCAgKMjiGKQT48FUIIKyPFLoQQVkaKXQgh\nrIxZRgrc8U6VSgFOlvmOS88buGh0iDIm79n62dr7hYr7nutprX1ut5AhxV5RKaWiizOnwZrIe7Z+\ntvZ+wfrfs5yKEUIIKyPFLoQQVkaK/c58bnQAA8h7tn629n7Byt+znGMXQggrI0fsQghhZaTYS0Ap\nNUEppZVS3kZnsTSl1PtKqYNKqT1KqWVKqSpGZ7IUpVQPpdQhpVSiUupFo/NYmlKqrlJqg1Jqv1Jq\nn1LqGaMzlRWllL1SKk4ptcroLJYgxX6HlFJ1gXuBU0ZnKSO/AEFa65bAYeAlg/NYhFLKHpgD9AQC\ngSFKqUBjU1lcATBBax0ItAXG2MB7/p9ngANGh7AUKfY79yEwCbCJDye01j9rrQuuffk7UPKRc+Vb\nBJCotT6mtc4DFgL9Dc5kUVrrc1rr2Gv/nIGp6OoYm8rylFK+QG/gC6OzWIoU+x1QSvUHzmitdxud\nxSCPAz8aHcJC6gCn//B1EjZQcv+jlPIHWgE7jE1SJj7CdHBWZHQQS5GxvX9xq4d3Ay9jOg1jVYrz\nwHKl1BRMv7pHlWU2YXlKKXdgKfCs1vqK0XksSSnVB7igtY5RSnU2Oo+lSLH/xc0e3q2UagEEALuv\nPT3GF4hVSkVorc+XYUSzu90Dy5VSjwF9gG7aeq+PPQPU/cPXvte+Z9WUUo6YSj1Ka/290XnKwN1A\nP6VUL8AF8FRKRWqthxucy6zkOvYSUkqdAMK11hVxkFCxKaV6ALOATlprq310jlLKAdOHw90wFfou\nYKjWep+hwSxImY5Q/gNc0lo/a3SesnbtiP0FrXUfo7OYm5xjF7czG/AAflFKxSulPjM6kCVc+4B4\nLLAW04eI31lzqV9zN/AI0PXav9v4a0eyooKTI3YhhLAycsQuhBBWRopdCCGsjBS7EEJYGSl2IYSw\nMlLsQghhZaTYhRDCykixCyGElZFiF0IIK/N/F57gkEeSKeIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fccac3be2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def huber(f,y):\n",
    "    rho = 2\n",
    "    loss = nd.abs(f-y)\n",
    "    loss = (loss > rho) * (loss - rho/2) + (0.5/rho) * (loss <= rho) * loss**2\n",
    "    return loss\n",
    "\n",
    "loss = huber\n",
    "with ag.record():    # start recording\n",
    "    theloss = loss(output, nd.zeros_like(output))\n",
    "theloss.backward()   # and compute the gradient\n",
    "\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='Huber Loss')\n",
    "plt.plot(output.asnumpy(), output.grad.asnumpy(), label='Gradient')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification - Logistic Regression\n",
    "\n",
    "Next consider the case where we have two labels, say ``cat`` and ``dog``. Since statisticians (and computers) don't like strings, we simplify this to $y \\in \\{\\pm 1\\}$. One way of mapping real numbers in $\\mathbb{R}$ into class probabilities is to use a sigmoid function.\n",
    "\n",
    "$$p(y|f) = \\frac{1}{1 + \\exp(-y f)} \\text{ and hence } -\\log p(y|f) = \\log(1 + \\exp(-y f))$$\n",
    "\n",
    "*Side remark for math nerds:* To keep the term numerically stable we can rewrite it as $-yf + \\log(1 + \\exp(yf))$ whenever $yf < 0$. The reason for doing this is to avoid exponentiating a large positive number which would trigger a numerical overflow.  Combining both expressions we get the following expression: $\\log(1 + \\exp(-|yf|)) - \\delta(yf < 0) \\cdot yf$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4FFXbx/HvyWbTG4SEFkJTahoQSOhNUBFRUSx0pIoC\nFnjhsWBXeB5FFEE6KKKCDQso0osgkEDAgKEHSCgpkN42m3n/2BCK9Gwyyeb+XNdcszs7mbkXwo+z\nZ2fOUZqmIYQQwnbY6V2AEEII65JgF0IIGyPBLoQQNkaCXQghbIwEuxBC2BgJdiGEsDES7EIIYWMk\n2IUQwsZIsAshhI2x1+OkVapU0erUqaPHqYUQotyKjIxM0jTN52b76RLsderUISIiQo9TCyFEuaWU\nOnEr+0lXjBBC2BgJdiGEsDES7EIIYWN06WMXpcdkMhEXF0dOTo7epQgb5OTkhJ+fH0ajUe9SxGUk\n2G1cXFwc7u7u1KlTB6WU3uUIG6JpGsnJycTFxVG3bl29yxGXsUqwK6VigXTADORrmhZqjeOK4svJ\nyZFQFyVCKYW3tzeJiYl6lyKuYs0We2dN05KseDxhJRLqoqTI71bZVK6+PN1z8gKzNx3VuwwhhLgj\nEWcjKI3pSK0V7Brwh1IqUik14lo7KKVGKKUilFIRd/rR7cc98Uz5LYYFW48Xp1ZRytzc3Ip9jNOn\nT/PYY49d9/WUlBRmzZp1y/tfbfDgwXz33XfFqlGIG/nywJcMWT2E1bGrS/xc1gr2dpqmNQfuB55V\nSnW4egdN0+ZqmhaqaVqoj89N74i9psk9m3B/QDXe/vUAyyNOFbNkUZ7UqFHjhsF7dbDfbH8hStOK\nIyuYumsq9/jfwz217ynx81kl2DVNiy9cJwA/Aq2scdyr2RvsmP5kCO3vrsKk7/fxe/SZkjiNKAWx\nsbF06dKFoKAgunbtysmTJwE4evQo4eHhBAYG8uqrrxa19mNjYwkICABg//79tGrVipCQEIKCgjh8\n+DCTJk3i6NGjhISEMGHChCv2N5vNjB8/noCAAIKCgpgxY8YNa1u3bh3NmjUjMDCQp59+mtzcXAAm\nTZpEkyZNCAoKYvz48QB8++23BAQEEBwcTIcO/2rPCMG6E+t4fdvrhFcPZ2qHqdjblfzFiMU+g1LK\nFbDTNC298HF34K1iV3YdjvYG5gxowYAFOxn7dRQLBtvT/u47+wRQ0bz5y34OnE6z6jGb1PDg9Qeb\n3vbPjRkzhkGDBjFo0CAWLlzI2LFjWbFiBePGjWPcuHE89dRTzJ49+5o/O3v2bMaNG0e/fv3Iy8vD\nbDYzZcoUoqOjiYqKAiz/EVw0d+5cYmNjiYqKwt7envPnz1+3rpycHAYPHsy6deto0KABAwcO5LPP\nPmPAgAH8+OOPxMTEoJQiJSUFgLfeeovVq1dTs2bNom1CXLT99HYmbJ5AQJUAPu78MQ4Gh1I5rzVa\n7FWBrUqpvcBOYKWmab9b4bjX5eJgz8JBLann48qILyKJPHGhJE8nSsD27dvp27cvAAMGDGDr1q1F\n2/v06QNQ9PrVWrduzXvvvcfUqVM5ceIEzs7ONzzX2rVrGTlyJPb2lnZM5cqVr7vvwYMHqVu3Lg0a\nNABg0KBBbN68GU9PT5ycnBg6dCg//PADLi4uALRt25bBgwczb948zGbzbfwJCFu3N3Ev4zaMo45n\nHWZ1nYWL0aXUzl3sFrumaceAYCvUcls8XYwsGRpGn9nbGLJoJ8tGtqZxdY/SLqNcuZOWdVnUt29f\nwsLCWLlyJT169GDOnDnUq1evRM9pb2/Pzp07WbduHd999x2ffvop69evZ/bs2ezYsYOVK1fSokUL\nIiMj8fb2LtFaRNl36MIhnln7DFWcqzC321w8HT1L9fzl6nLHq/m4O/LlsDBcHe0ZsGAnx5My9S5J\n3KI2bdrwzTffALB06VLat28PQHh4ON9//z1A0etXO3bsGPXq1WPs2LE89NBD7Nu3D3d3d9LT06+5\nf7du3ZgzZw75+fkAN+yKadiwIbGxsRw5cgSAJUuW0LFjRzIyMkhNTaVHjx589NFH7N27F7B8JxAW\nFsZbb72Fj48Pp07Jl/oV3cm0k4xcMxJne2fmdZ9HFecqpV5DuQ52AL9KLiwZGkaBptF//g7OpGbr\nXZK4SlZWFn5+fkXLtGnTmDFjBosWLSIoKIglS5bw8ccfAzB9+nSmTZtGUFAQR44cwdPz3y2d5cuX\nExAQQEhICNHR0QwcOBBvb2/atm1LQEAAEyZMuGL/YcOG4e/vT1BQEMHBwXz11VfXrdXJyYlFixbR\np08fAgMDsbOzY9SoUaSnp9OzZ0+CgoJo164d06ZNA2DChAkEBgYSEBBAmzZtCA4u9Q+vogw5l3mO\nEWtGkF+Qz7xu86jpVlOXOlRpXCx/tdDQUM3aE21Ex6fy1Ny/8PVwZPnI1ni7OVr1+OXVP//8Q+PG\njfUu45ZlZWXh7OyMUopvvvmGr7/+mp9++knvssQNlLffsZJyIecCQ34fwtmssyzovoCmVazf9amU\niryVIVvKfYv9ooCaniwY3JK4C9kMWrSTtByT3iWJOxAZGVl0GeOsWbP48MMP9S5JiJvKyMvgmbXP\ncCr9FDO6zCiRUL8dNjW6Y6u6lZndvwXDv4hg2OIIPn+6Fc4OBr3LErehffv2Rf3XQpQHOfk5jN0w\nloPnDzK983RaVmupd0m202K/qHMjXz56IoRdJ84zemkkefkFepckhLBRpgITEzZNIOJsBO+0e4eO\ntTrqXRJgg8EO8GBwDd57JJANBxN5cXkU5oLS/x5BCGHbCrQCXvvzNTbGbeSVsFd4oN4DepdUxKa6\nYi73VCt/0rJNvP9bDO5ORt57JECGGBVCWIWmaby34z1WHlvJuObjeKLRE3qXdAWbDXaAkR3rk5pt\nYtbGo3g6G5l0fyO9SxJC2IAZe2aw7OAyhjQdwtCAoXqX8y822RVzuQn3NqR/uD+zNx1l1sYjepdT\nIZ07d46+fftSr149WrRoQevWrfnxxx+Ldcw33niDDz74AIDJkyezdu3aOzpOVFQUq1atuu2fe+qp\npwgKCuKjjz66bl23KyIigrFjx1739djY2Cuuwb/Z/lfr1KkT1r7MuCJaHL2YeX/P49G7H+WFFi+U\nyZ4Am26xg2WGl7d6BZCek89/fz+Ih5OR/uG19S6rwtA0jYcffphBgwYVhdKJEyf4+eef/7Vvfn5+\n0Xgut+Ott+58zLmoqCgiIiLo0aPHLf/M2bNn2bVrV9HdqdYSGhpKaOj1L1G+GOwXx9C52f7C+r4/\n9D0fRn7IvXXu5bXw18pkqEMFaLED2NkpPugTTNdGvrz2UzQ/RcXrXVKFsX79ehwcHBg1alTRttq1\nazNmzBgAFi9eTK9evejSpQtdu3YlIyODrl270rx5cwIDA6+4Oendd9+lQYMGtGvXjoMHDxZtv3yS\njMjISDp27EiLFi249957OXPGMrRzp06dmDhxIq1ataJBgwZs2bKFvLw8Jk+ezLJlywgJCWHZsmVX\n1J6Tk8OQIUMIDAykWbNmbNiwAYDu3bsTHx9PSEgIW7Zsue57j4qKIjw8nKCgIB555BEuXLAMVrdr\n1y6CgoKKhhi+OLzwxo0b6dmzJwCbNm0iJCSEkJAQmjVrRnp6OpMmTWLLli2EhITw0UcfXbF/RkZG\nUa1BQUFFwzJcz9dff110x+zEiRMBy/DGgwcPJiAggMDAwKJPI5988knRcMVPPvnkDY9ry1bHrubN\n7W/SrmY73m/3Pga7snsptc232C8yGuyY2a85gxbu5KXle3FztKdr46p6l1W6fpsEZ/+27jGrBcL9\nU6778v79+2nevPkND7F792727dtH5cqVyc/P58cff8TDw4OkpCTCw8Pp1asXu3fv5ptvviEqKor8\n/HyaN29OixYtrjiOyWRizJgx/PTTT/j4+LBs2TJeeeUVFi5cCFg+EezcuZNVq1bx5ptvsnbtWt56\n6y0iIiL49NNP/1XXzJkzUUrx999/ExMTQ/fu3Tl06BA///wzPXv2LBoi+HoGDhzIjBkz6NixI5Mn\nT+bNN99k+vTpDBkyhHnz5tG6dWsmTZp0zZ/94IMPmDlzJm3btiUjIwMnJyemTJnCBx98wK+//gpY\n/iO46O2338bT05O//7b8/V78T+RaTp8+zcSJE4mMjKRSpUp0796dFStWUKtWLeLj44mOjgYoGoZ4\nypQpHD9+HEdHxwo7NPHW+K1M2jKJZr7NmNZpGkaDUe+SbqhCtNgvcjIamD8olCY1PBi9dDd/HUvW\nu6QK59lnnyU4OJiWLS/dxNGtW7eioXQ1TePll18mKCiIe+65h/j4eM6dO8eWLVt45JFHcHFxwcPD\ng169ev3r2AcPHiQ6Oppu3boREhLCO++8Q1xcXNHrvXv3BqBFixZXjNd+PVu3bqV///4ANGrUiNq1\na3Po0KFbep+pqamkpKTQsaPluuaLw/+mpKSQnp5O69atgesPTdy2bVtefPFFPvnkE1JSUm7aRbV2\n7VqeffbZoueVKlW67r67du2iU6dO+Pj4YG9vT79+/di8eTP16tXj2LFjjBkzht9//x0PD8toqUFB\nQfTr148vv/zyjrrKyrvd53bzwoYXuMvrLmZ0nYGz/Y2HiS4LKtzfkruTkcVDWvHEnO0M+zyCr4aH\nEeTnpXdZpeMGLeuS0rRp0yu6BWbOnElSUtIVfcOurq5Fj5cuXUpiYiKRkZEYjUbq1KlDTk7OLZ1L\n0zSaNm3K9u3br/m6o6Nl/CCDwVA00mNZNWnSJB544AFWrVpF27ZtWb265OfJrFSpEnv37mX16tXM\nnj2b5cuXs3DhQlauXMnmzZv55ZdfePfdd/n7778rTMDHnI/huXXPUc21GrPvmY2HQ/kYGrxCtdgv\nquzqwJKhYXi5GBm0cCeHz117uFdRfF26dCEnJ4fPPvusaFtWVtZ1909NTcXX1xej0ciGDRs4ceIE\nAB06dGDFihVkZ2eTnp7OL7/88q+fbdiwIYmJiUXBbjKZ2L9//w3ru9Fwv+3bt2fp0qUAHDp0iJMn\nT9KwYcMbv+FCnp6eVKpUqagP/uLwv15eXri7u7Njxw7g+kMTHz16lMDAQCZOnEjLli2JiYm56dDE\nM2fOLHp+o66YVq1asWnTJpKSkjCbzXz99dd07NiRpKQkCgoKePTRR3nnnXfYvXs3BQUFnDp1is6d\nOzN16lRSU1PJyMi4pT+D8i42NZaRa0bi6uDK3G5z8XYuP+PsV8hgB6jm6cTSYWHYG+zov2AHp85f\nP2zEnVNKsWLFCjZt2kTdunVp1aoVgwYNYurUqdfcv1+/fkRERBAYGMgXX3xBo0aWew+aN2/OE088\nQXBwMPfff/8VXTkXOTg48N133zFx4kSCg4MJCQlh27ZtN6yvc+fOHDhw4Jpfno4ePZqCggICAwN5\n4oknWLx4cVGr/1Z8/vnnTJgwgaCgIKKiopg8eTIACxYsYPjw4YSEhJCZmXnNoYmnT59eNEer0Wjk\n/vvvJygoCIPBQHBw8L8us3z11Ve5cOFC0fyrF7/ovZbq1aszZcoUOnfuTHBwMC1atOChhx4iPj6e\nTp06ERISQv/+/Xn//fcxm83079+/6AvksWPH4uVl+59wz2aeZfia4QDM6zaP6m7Vda7o9tjMsL13\n6uDZdB6fsx0vFyPfjmyNr4eT3iVZlQypWvZkZGQUTdI9ZcoUzpw5UzQefXlka79jydnJDP59MMnZ\nySy4dwGNvcvOe6tww/beqYbV3Fk8pCWJ6bkMWLCTlKw8vUsSNm7lypWEhIQQEBDAli1bePXVV/Uu\nSRRKz0vnmbXPcDbzLDPvmVmmQv12VPhgB2jmX4l5A0M5npTJ4EW7yMwt21+sifLtiSeeICoqiujo\naFauXImPj4/eJQkgOz+b59Y9x+GUw3zU+SOa+TbTu6Q7JsFeqO1dVZjRtxl/x6cyYkkEOSaZcV6I\nisJkNvHixheJSoxiSvsptKvZTu+SikWC/TL3Nq3Gfx8N4s8jyYz9eg/5ZhnLXQhbZy4w85+t/2Fr\n/FYmh0/m3jr36l1SsUmwX+XRFn688WAT/jhwjonf/02BjOUuhM0q0Ap4c/ubrI5dzfjQ8Tza4FG9\nS7KKinGXwW0a3LYuaTn5TFtzCHcne15/sEmZHexHCHFnNE3jf7v+x49HfmRU8CgGNR2kd0lWY7UW\nu1LKoJTao5T61VrH1NOYLncxrF1dFm+LZfraw3qXU65dvLSvOE6fPs1jjz123ddTUlKYNWvWLe9/\ntcsHErO2CRMm0LRpUyZMmFAix79VMTExtG7dGkdHxzseWtiWzNo7iy//+ZL+jfszOni03uVYlTVb\n7OOAf4Dycc/tTSileOWBxqTlmPh43WE8nI0MbVdX77IqrBo1atwweC8G++jRo29p/9I0d+5czp8/\nj8Fwa6MB3unwxTdTuXJlPvnkE1asWGH1Y5c3i6MXM3vvbHrf3Zv/a/l/NveJ3CotdqWUH/AAMN8a\nxysrlFK83zuIHoHVePvXAyyPOKV3STYjNjaWLl26EBQURNeuXTl58iRguZU+PDycwMBAXn311aLW\nfmxsbNHwtvv376dVq1aEhIQQFBTE4cOHmTRpEkePHi0aCvfy/c1mM+PHjy+6k3PGjBk3rG3dunU0\na9aMwMBAnn76aXJzcwHL+C0Xh68dP348AN9++23R3Z4dOnT417F69epFRkYGLVq0YNmyZdd934MH\nD2bUqFGEhYXxf//3f1cco0OHDleMJNmuXTv27t1723/mvr6+tGzZEqOxbI9MWNK+PfRt0Zjqk8Mn\n21yog/Va7NOB/wPcr7eDUmoEMALA39/fSqcteQY7xUdPhJCRG8mk7/fh6mDPA0Hl6/bii6bunErM\n+RirHrNR5UZMbDXxtn9uzJgxDBo0iEGDBrFw4ULGjh3LihUrGDduHOPGjeOpp55i9uzZ1/zZ2bNn\nM27cOPr160deXh5ms5kpU6YQHR1dFICXj944d+5cYmNjiYqKwt7envPnz1+3rpycHAYPHsy6deto\n0KABAwcO5LPPPmPAgAH8+OOPxMTEoJQqGr72rbfeYvXq1dSsWfOaQ9r+/PPPuLm5FdX14IMPXvN9\nA8TFxbFt27Z/teyHDh3K4sWLmT59OocOHSInJ6do2IAXXnjhX+d0cXG56VAKFdXKYyt5e/vbdPDr\nUObHVC+OYrfYlVI9gQRN0yJvtJ+maXM1TQvVNC20vN2Q4WhvYHb/5rSoXYlx3+xhQ0yC3iWVe9u3\nby8asnbAgAFs3bq1aHufPn2A6w9p27p1a9577z2mTp3KiRMncHa+8TCqa9euZeTIkUXdGxeHCL6W\ngwcPUrduXRo0aABcGm7X09MTJycnhg4dyg8//ICLiwtgGV538ODBzJs3D7P55vc+XO99A/Tp0+ea\n3TV9+vTh119/xWQysXDhQgYPHgxYxrmJior61yKhfm0bTm7gla2vEFotlA87fljmx1QvDmu02NsC\nvZRSPQAnwEMp9aWmaf2tcOwyw8XBngWDW9Jv3g5GfRnJ4iGtaF2//Iz2BtxRy7os6tu3L2FhYaxc\nuZIePXowZ84c6tWrV6LntLe3Z+fOnaxbt47vvvuOTz/9lPXr1zN79mx27NjBypUradGiBZGRkXh7\n39nvxeXDF1/OxcWFbt268dNPP7F8+XIiIy1tqJu12GfOnMm8efMAWLVqFTVq1LijumzBX2f+Yvym\n8TTxbsKMLjNwsretMaGuVuwWu6Zp/9E0zU/TtDrAk8B6Wwv1izycjHz+dCv8K7sw7PNd7Dl5/aFR\nxY21adOmaMjapUuX0r59ewDCw8OLxm+/3pC2x44do169eowdO5aHHnqIffv23XRI2zlz5hSNwX6j\nrpiGDRsSGxtbNJ/pxeF2MzIySE1NpUePHnz00UdFfdxHjx4lLCyMt956Cx8fH06duvH3MNd73zcz\nbNgwxo4dS8uWLYsm0bhZi/3ZZ58t2laRQz0qIYqx68dS27M2n93zGa7Ga/8HakvkBqXbVNnVgaXD\nwqji7sighTs5cDpN75LKvKysLPz8/IqWadOmMWPGDBYtWkRQUBBLliwpGt1w+vTpTJs2jaCgII4c\nOXLNIW2XL19OQEAAISEhREdHM3DgQLy9vWnbti0BAQH/uqxw2LBh+Pv7ExQURHBwcNGk2tfi5OTE\nokWL6NOnD4GBgdjZ2TFq1CjS09Pp2bMnQUFBtGvXjmnTpgGWSxkvzh3apk0bgoODb/hncb33fTMt\nWrTAw8ODIUOG3NL+13L27NmiP/933nkHPz8/0tJs+/c35nwMo9eNxsfZh7nd5uLp+O/fJ1tU4Yft\nvVNxF7LoM3s7efkFLB/Vmvo+xb9WuySUtyFVs7KycHZ2RinFN998w9dff33FhNYV1enTp+nUqRMx\nMTHY2ZWt9lhZ/R07nnqcwb8PxsHgwOf3fU4Nt/L/qUWG7S1hfpVcWDosDKWg/3yZqMNaIiMjiy5j\nnDVrFh9++KHeJenuiy++ICwsjHfffbfMhXpZFZ8Rz/A/Lk2UYQuhfjukxV5M/5xJ48m5f+HpbOTb\nUa2pWsYm6iirrSlhO8ra71hiViKDfh9ESm4Ki+5dRMPKtzadYXkgLfZS0ri6B58/3YrkjFz6z9/B\n+cyyN1GHHv95i4qhrP1upeSkMGLNCJKyk5h9z2ybCvXbIcFuBSG1vFgwuCUnz2cxcOEO0nJMepdU\nxMnJieTk5DL3D1CUf5qmkZycjJNT2fiUmpGXwai1oziZdpJPu3xKkE+Q3iXpRrpirGjDwQRGfBFB\nsJ8XXwxthYuD/oNnmkwm4uLiyMnJ0bsUYYOcnJzw8/PTfZiC7Pxsnln7DHsT9jK983Q61uqoaz0l\n5Va7YiTYrey3v8/w7Fe7aVO/CvMHheJktM1bloUoK0xmE2M2jGFb/Db+2+G/3Ff3Pr1LKjHSx66T\n+wOr89/Hgtl6JInnvtqDSWZhEqLE5BfkM3HLRP6M/5PXW79u06F+OyTYS8BjLfx4+6GmrP3nHC8t\n34tZZmESwuoKtALe2PYGa06sYULoBJuZ/cga9O8EtlEDWtchI9fM1N9jcHEw8H7vQJscHlQIPWia\nxtSdU/np6E+MDh7NwKYD9S6pTJFgL0HPdKpPZm4+n244gqujPa8+0FjCXQgrmLFnBl/FfMXAJgMZ\nFTxK73LKHAn2EvZS9wZk5OazYOtxXB3tebFbA71LEqJcm7tvLvP+nsejdz/K+NDx0li6Bgn2EqaU\nYnLPJmTl5fPJusO4ORoY0aG+3mUJUS59vv9zZuyZwYP1HmRya9uc/cgaJNhLgZ2dZYq9rDwz762K\nwcXBnv7htfUuS4hyZVnMMj6I+IButbvxVtu3sFNy7cf1SLCXkotT7OWYzLz2UzSujgYeaeand1lC\nlAs/HfmJd3a8Q0e/jkxtPxV7O4muG5H/8kqR0WDHp32b06a+N+O/3cfv0Wf1LkmIMu/3478zedtk\nwquH82En257Szlok2EuZk9HA3AGhhNTyYszXu9lwUOZPFeJ61p9cz3+2/IcQnxA+7vwxjgZHvUsq\nFyTYdeDqaM/CwS1pWM2dUUsi+fNIkt4lCVHm/Bn/J+M3jaexd2Nmdp2Ji9FF75LKDQl2nXg6G1ny\ndBh1q7gy7PMIdh6//jycQlQ0u87uYtyGcdT3qs9n93yGm0PZnKGsrJJg11ElVweWDA2jhpcTQxbt\nZLdMji0EUQlRPLvuWWq61WROtzkVZp5Sa5Jg15mPuyNfDQ8vmhw7Oj5V75KE0M2B5AOMXmuZfHp+\n9/lUdqqsd0nlkgR7GVDVw4mvhofj4WSk/4IdxJy17ZnjhbiWwxcOM3LNSNwc3JjffT4+Lj56l1Ru\nSbCXETW9nPl6eDhO9gb6zdvBkYR0vUsSotTEpsYy/I/hGO2MLOi+gOpu1fUuqVyTYC9D/L1d+Gp4\nGEop+s7bQWxSpt4lCVHi4tLjGPbHMDQ05nefTy2PWnqXVO5JsJcx9Xzc+Gp4GPkFGv3m7yDuQpbe\nJQlRYs5mnmXYH8PIys9ibre51POqp3dJNqHYwa6UclJK7VRK7VVK7VdKvWmNwiqyBlXdWTK0Fek5\nJvrO28GZ1Gy9SxLC6pKykxj+x3BSclOYc88cGlZuqHdJNsMaLfZcoIumacFACHCfUircCset0JrW\n8GTJ0DDOZ+bRb94OEtJlMmphO87nnGf4H8M5l3WOmV1nEugTqHdJNqXYwa5ZZBQ+NRYuMhecFQTX\n8mLxkJacTcuh37wdJGfk6l2SEMWWkpPC8D+Gcyr9FDO6zKBF1RZ6l2RzrNLHrpQyKKWigARgjaZp\nO66xzwilVIRSKiIxMdEap60QQutUZsGglpw8n8WABTtJycrTuyQh7lhqbioj1owgNjWWTzp/Qlj1\nML1LsklWCXZN08yapoUAfkArpVTANfaZq2laqKZpoT4+cn3q7Whd35u5A0M5kpDBoIU7Scsx6V2S\nELctLS+NkWtGciTlCNM7T6dNzTZ6l2SzrHpVjKZpKcAG4D5rHldAxwY+zOrXnP2n0xiyaBeZufl6\nlyTELcvIy+CZNc9w8MJBpnWaRnu/9nqXZNOscVWMj1LKq/CxM9ANiCnuccW/3dOkKjOeakbUqRSG\nfr6L7Dyz3iUJcVNZpixGrxvN/uT9fNDhAzrV6qR3STbPGi326sAGpdQ+YBeWPvZfrXBccQ33B1Zn\n2uPB7Dh+nuFfRJBjknAXZdfFUN+XuI+pHabStXZXvUuqEIo9v5SmafuAZlaoRdyih0JqYjJrTPhu\nL8O/iGDewFCcjAa9yxLiCtn52YxdP5Y9CXt4v9373FvnXr1LqjDkztNy6rEWfvz30SC2HklixJJI\nabmLMiXXnMu49ePYeXYn77R9hx71euhdUoUiwV6O9QmtxdTeQWw+lMhICXdRRuSZ83h+w/NsP7Od\nN9u8yYP1H9S7pApHgr2ce7xlLab0DmTToUSe+TKS3HwJd6Efk9nESxtfYmv8Vl5v/TqP3P2I3iVV\nSBLsNuDJVv683zuQDQcTeebL3RLuQhemAhMTNk9gY9xGXgl7hccaPKZ3SRWWBLuNeKqVP+89Esj6\nmASeXSpJbZzIAAAgAElEQVThLkpXfkE+kzZPYt3JdUxqNYknGz2pd0kVmgS7Dekb5s87Dwew9p8E\nnl26h7z8Ar1LEhWAqcDExM0T+ePEH4wPHU+/xv30LqnCk2C3Mf3Da/P2Q01Z+885nv1qt4S7KFFX\nh/qgpoP0LkkgwW6TBrSuw5u9mrLmwDnGfL0bk1nCXVifyWxiwqYJrDmxhgmhEyTUyxAJdhs1qE0d\nXn+wCav3n2Ps13sk3IVVmcwmxm8az7qT6/i/lv/HwKYD9S5JXEaC3YYNaVuX13o24bfos4z7RsJd\nWIfJbOKlTS+x/tR6JrWaxIAmA/QuSVyl2EMKiLJtaLu6aJrGOyv/Afbw8ZPNMBrk/3NxZ/LMeby0\n8SU2xm3kP63+Q9/GffUuSVyDBHsFMKy9ZYLgd1b+Q755N5/2bY6DvYS7uD155jxe3Pgim+I28XLY\nyzzV6Cm9SxLXIf+6K4hh7evxZq+m/HHgHKO+lOEHxO3JNefy/Ibn2RS3iVfDXpVQL+Mk2CuQQW3q\n8O4jAayPSZCBw8QtuxjqW+K38Fr4azzR6Am9SxI3IcFewfQLq81/Hw1iy+FEnl68i6w8mYlJXF+u\nOZdxG8axNX4rk1tP5vGGj+tdkrgFEuwV0OMta/Fhn2D+OpbM4EW7yJBp9sQ1XBxP/c/4P3mj9Rv0\nadBH75LELZJgr6B6N/fjoydCiDxxgUELd5IuE2SLy2SaMhm9djTbT2/nrTZv8WiDR/UuSdwGCfYK\n7KGQmsx4qhl7T6XQf8FOUrMl3AWk5aUxYs0I9iTsYUr7KTL0bjkkwV7B9Qiszqx+zTlwOpX+83eQ\nkpWnd0lCRyk5KQxbPYwDyQf4oOMHMvNROSXBLujetBpzB4Ry8Fw6T83bwflMCfeKKCk7iSGrh3A0\n5Sgfd/6Ye2rfo3dJ4g5JsAsAOjfyZf7AUI4lZvDk3O0kpOXoXZIoRWczzzLk9yHEZ8Qz856ZdPDr\noHdJohgk2EWRDg18WDSkJXEXsukzZzunzmfpXZIoBfEZ8Qz+fTCJ2YnMvmc24dXD9S5JFJMEu7hC\nm/pV+HJYGBcy83h8znaOJmboXZIoQSfSTjDot0Gk56Uzv/t8mldtrndJwgok2MW/NPevxLKRrTGZ\nC3h89nb2n07VuyRRAo6mHGXw74PJM+ex8N6FBFQJ0LskYSXFDnalVC2l1Aal1AGl1H6l1DhrFCb0\n1bi6B8tHtsbR3o4n5/5F5InzepckrGh/0n4G/z4YhWLRfYtoWLmh3iUJK7JGiz0feEnTtCZAOPCs\nUqqJFY4rdFbPx41vn2mDt6sD/efvZOvhJL1LElaw88xOnl79NK5GVxbft5j6XvX1LklYWbGDXdO0\nM5qm7S58nA78A9Qs7nFF2VDTy5nlo1pT29uFpxfv4o/9Z/UuSRTD+pPreWbtM1R3rc7n932Ov4e/\n3iWJEmDVPnalVB2gGbDjGq+NUEpFKKUiEhMTrXlaUcJ83Z34ZkQ4TWp48MzS3azYE693SeIO/Hz0\nZ17c+CINKzdk8X2LqepaVe+SRAmxWrArpdyA74HnNU1Lu/p1TdPmapoWqmlaqI+Pj7VOK0qJl4sD\nXw4Lo1WdyrywPIov/zqhd0niNnx54Ete2foKodVCmd99Pl5OXnqXJEqQVYJdKWXEEupLNU37wRrH\nFGWPm6M9i4a0pEtDX15dEc2MdYfRNE3vssQNaJrGrKhZTN01lXv872FW11m4GF30LkuUMGtcFaOA\nBcA/mqZNK35JoixzMhqYPaAFvZvV5MM1h3j95/2YCyTcy6ICrYApO6fw2d7PePiuh/lfx//hYHDQ\nuyxRCqwx52lbYADwt1IqqnDby5qmrbLCsUUZZDTY8UGfYKq4OzJ38zGSM/KY9kQwjvYGvUsThUwF\nJib/OZlfj/3KoCaDeCn0JSxtMFERFDvYNU3bCshvTAVjZ6d4uUdjqrg58N6qGC5k5TFnQAvcnYx6\nl1bhZZoyeWnjS/x5+k/GNhvLsMBhEuoVjNx5KoplRIf6THs8mJ3Hz/PEnL9ISJfBw/SUlJ3E06uf\n5q8zf/FmmzcZHjRcQr0CkmAXxda7uR/zBoVyPCmTxz7bzonkTL1LqpBOpJ1gwKoBHE89ziddPqH3\n3b31LknoRIJdWEXnhr58NTyM9BwTj362jeh4GV+mNP2d+DcDVg0g05TJgu4LZNjdCk6CXVhNM/9K\nfDuqDQ4GO56Ys52NBxP0LqlC2By3maF/DMXF6MKSHksI9AnUuyShMwl2YVV3+brxw+i2+Hu7MvTz\nCL7acVLvkmzaD4d/YOz6sdTxqMOXPb6ktkdtvUsSZYAEu7C6ap5OfDuqNe3uqsLLP/7N1N9jKJBr\n3a1K0zQ+i/qM17e9Tnj1cBbdt4gqzlX0LkuUERLsokS4OdqzYFAofcP8+WzjUcZ+s4cck1nvsmxC\nnjmPl7e+zKy9s+hVvxczus7A1eiqd1miDLHGDUpCXJO9wY53Hw7Av7ILU36L4WxqDvMGhlLJVe5+\nvFMXci7w/Ibn2Z2wmzHNxjA8UC5nFP8mLXZRopRSjOpYn0/7NmNffCq9P9tGbJJcDnknjqUeo9+q\nfkQnRfO/Dv9jRNAICXVxTRLsolT0DKrBV8PCSMnKo/dn29h5XGZkuh07zuyg/6r+lssZ713AfXXv\n07skUYZJsItSE1qnMj+MbouXs5F+8/9i2S65YuZW/Hj4R0atGYWvsy9LeywlxDdE75JEGSfBLkpV\n3Squ/Di6LeH1vJn4/d+8+ct+8s0FepdVJpkLzEyLnMbkbZNpWa0lS3oswc/dT++yRDkgwS5KnaeL\nkUWDW/J027os+jOWIYt3kZpl0rusMiUtL43n1j/HouhFPN7gcWbeMxN3B3e9yxLlhAS70IW9wY7J\nDzbhv48G8dexZB6Z9SdHEzP0LqtMOJZyjL4r+/LX6b94Lfw1Xmv9GkY7GTVT3DoJdqGrx1vW4qvh\n4aRmm3h45p9sOlSx58PdeGojfVf1JT0vnQX3LuDxho/rXZIohyTYhe5a1qnMT8+1paaXM0MW7WTm\nhiMV7k5VTdOYu28uY9ePxd/dn2U9l9G8anO9yxLllAS7KBP8Krnww+g2PBBUg/+tPsiIJZGkZleM\nfvdMUyYvbXqJGXtm0KNeD764/wuquVbTuyxRjkmwizLDxcGeT54M4fUHm7DxYAIPfbqVmLNpepdV\noo5cOMJTK59i3cl1vNTiJd5v9z5O9k56lyXKOQl2UaYopRjSti5fjwgnM8/MIzO38VNUvN5llYhf\nj/1K31V9SctNY373+QwOGCx3kgqrkGAXZVLLOpVZOaYdgTU9GfdNFG/8vJ+8fNu43j3XnMvb29/m\nP1v+QxPvJnz74Le0rNZS77KEDZFgF2WWr4cTS4eHMbRdXRZvi6XP7G2cTM7Su6xiic+IZ+BvA1l+\naDlDAoYwv/t8fFx89C5L2BgJdlGmGQ12vNazCbP7N+d4UiYPfLKFX/ae1rusO7L2xFoe/+VxTqWd\n4uPOH/Niixext5MBVoX1SbCLcuG+gOqsGteeu6u6MebrPUz6fh/ZeeVjfPcsUxZvbHuDFza+QC33\nWizruYwu/l30LkvYMAl2UW74VXJh2cjWPNu5PssiTvFgObhqJuZ8DE+ufJIfDv/A0wFPs+T+JdTy\nqKV3WcLGWSXYlVILlVIJSqloaxxPiOsxGuyYcG8jljwdRkqWiYc+/ZPPt8WWuRuaNE1jyYEl9F3Z\nl4y8DOZ2n8sLLV7AaJChAUTJs1aLfTEgA0SLUtPu7ir8Nq49ret78/rP+xm4cCenU7L1LguAhKwE\nnln3DP/d9V/a1mzL972+J7x6uN5liQrEKsGuadpmQGZOEKXKx92RRYNb8t4jgew+eYF7p2/mh91x\naJo+rXdN0/jl6C88/NPDRJ6N5JWwV/ik8ydUcqqkSz2i4iq1Pnal1AilVIRSKiIxsWIP9CSsRylF\n3zB/fhvXnkbV3Hlx+V6e+XI3yRm5pVpHUnYSz294npe3vkw9z3p81+s7nmz0pNxwJHShrNW6UUrV\nAX7VNC3gZvuGhoZqERERVjmvEBeZCzTmbznGh38cwsPZnrceCuD+gGolHq6rY1fz7l/vkmnK5Llm\nzzGwyUAMdoYSPaeomJRSkZqmhd5sP7kqRtgMg51iZMf6/DymLdU8nRi9dDfDv4jkTGrJ9L0nZCXw\n4sYXGb9pPDXcarD8QctNRxLqQm8S7MLmNKrmwYrRbXmlR2O2Hkmk27TNLPnrhNWunCnQCvgm5ht6\nrejF5rjNjGs+ji97fEl9r/pWOb4QxWWtyx2/BrYDDZVScUqpodY4rhB3yt5gx/AO9fjj+Y6E1PLi\ntRXRPD5nO4fPpRfruIcuHGLAbwN4d8e7BFYJ5IdePzAscJjcQSrKFKv1sd8O6WMXpUnTNH7YHc/b\nKw+QkZPPkLZ1GNv1btydbv2a8ixTFnP2zeGL/V/g7uDOhJYT6Fmvp3w5KkrVrfaxSzND2DylFI+2\n8KNTQx/+t/og87ceZ0XUaf5zfyMeaVbzhuFcoBWw8thKPor8iMTsRB6q/xDjQ8fj5eRViu9AiNsj\nLXZR4ew9lcLkn/ez91QKobUr8UavpgTU9PzXftFJ0by/8332Je4jwDuAia0mEuIbokPFQljcaotd\ngl1USAUFGt/tjmPqbzGcz8rj0eZ+vNitATW8nEnISuCT3Z/w09Gf8HbyZlzzcTx010PYKbnWQOhL\ngl2IW5CabeLT9Yf5fNsJlCGbkMA9HM39HbOWz4DGAxgRNAI3Bze9yxQCkD52IW6Jp7ORl+6th6vv\nFj4/sIgDWZmQ0YyBjUYyOjgcJ6Ncky7KHwl2UWHlmfNYcWQFc/bNISErgXY12/FgraF882cBM9ck\n8v2OjYzuXJ/HQ2tJwItyRbpiRIWTk5/D94e/Z2H0QhKyEgjyCeL55s9fMe/otiNJfLT2ELtiL1DV\nw5FnOtbnyVb+EvBCV9LHLsRVskxZLD+4nMX7F5Ock0xz3+aMDB5J6+qtr3nJo6ZpbD+azPR1h9l5\n/Dy+7o6M6FCPJ1v54+YoH3ZF6ZNgF6LQucxzfB3zNd8e+pa0vDTCqocxMmjkFS30m9l+NJmP1x3i\nr2PncXeyp2+YP0Pa1KWap1MJVi7ElSTYRYW3P3k/Sw4sYfXx1RRQQFf/rgxsMrBY16LvOXmB+VuO\n81v0GeyUoldwDYa2r0vTGv++Dl4Ia5NgFxVSrjmXNSfW8O3Bb9mdsBsXexd6392bfo374efuZ7Xz\nnDqfxcI/j7Ns1ymy8sy0qF2JfmH+9AisLv3wosRIsIsK5WjKUb479B2/HPuF1NxUarnX4omGT9D7\n7t64O7iX2HlTs0x8G3mKr3ac5FhSJl4uRh5r7sdTYf7U95Hr34V1SbALm5eWl8aa2DX8fPRndifs\nxt7Onq7+XXmswWO0qtaqVO8U1TSN7ceSWbrjJKujz5JfoNGyTiUeblaTnoE18HSRSaxF8UmwC5uU\na85l06lNrDq+is1xmzEVmKjjUYfed/emV/1eeDt7610iCek5fBcZxw+74zmSkIGDwY4ujXx5uFlN\nOjfywdFeumrEnZFgFzYjOz+b7ae3s+7kOtafXE+GKYMqzlW4r859PFDvAZp6Ny2Tw+dqmsb+02n8\nsDuen/eeJikjFw8ne7o2rsq9TavRsYEPzg4S8uLWSbCLci0lJ4VNcZtYf3I9205vI8ecg7uDO51r\ndeaBeg8QVi2sXE1Bl28u4M+jyfwcdZp1MedIyTLhZLSjUwNf7g2oSueGvni5OOhdpijjZKwYUa7k\nF+QTnRTN9jPb+ev0X0QlRlGgFVDVpSqP3P0IXfy70KJqC4x25bOv2t5gR8cGPnRs4EO+uYCdx8/z\n+/6zrN5/lt/3n8VOQZCfFx0b+NChgQ8htbww2JW9TyGifJAWu9CFpmkcTztOxNkItp3exs4zO0k3\npaNQNPFuQtuabenq35XGlRuXyW4Wayko0Ngbl8LGg4lsPpzI3lMpFGiWwcna3VWF8HqVaVXXm7t9\n3bCToK/wpCtGlCkms4n9yfvZk7CnaEnJTQGgumt12tRoQ+sarQmrFlahZydKycpj65EkNh1MZMvh\nJM6m5QDg5WIktHZlWtWtRKu63jSp7oGDvYwPX9FIsAvdmApMHEs5xoHkA/xz/h8OJB8g5nwMueZc\nAGp71KaZbzOa+zanedXm+Lv723Sr/E5pmsap89nsjD3PzuPJ7Iq9wPGkTAAcDHY0ru5OoJ8nQX5e\nBPt5cZevm3Tf2DgJdlHiNE0jOSeZYynHOJp6lCMXjnAg+QCHLhwiryAPABd7FxpVbkRAlQCa+zYn\n2DeYKs5VdK68/EpIy2FX7AX2xaWwLy6Vv+NTycjNB8DZaKBxdXcaVnOnQVV3GlZ1p0E1d6q4Oepc\ntbAWCXZhNdn52ZzOOE1cehwn009yNOUox1KPcSz1GKm5qUX7uRvdaeTdiCaVm9DEuwmNvRtT26O2\nTClXggoKNI4nZ7IvLoW9p1KJOZvGwbPpXMgyFe1T2dWBBlXdqOfjRh1vF2p7u1LH2xX/yi5yuWU5\nI8EubommaWSYMkjMSuRc1jnOZZ0jPiOeuPQ4y5IRR1J20hU/4+noSX3P+tT3siz1POtR36s+Ps4+\n0qVSBmiaRlJGHofOpXPwbDqHzlmW40mZVwQ+QDUPJ2p7u1Crsgs1PJ2o7uVMdU8nang5U83TCQ+n\n8nkVkq2Syx0rME3TyM7P5kLuBVJyUriQe4ELOZYlKSeJhKwEErISisI8Oz/7ip+3U3ZUdamKn7sf\n7Wu2x8/dDz83P8va3Y9KjpUkwMswpRQ+7o74uDvS9q4ru71Ss0ycOJ9JbHIWJ5Is69jkTLYcTiQh\nPZer23lujvZU93SimqcTVdwc8XZ1oIp74drNEW83y7qyq4MMflaGWCXYlVL3AR8DBmC+pmlTrHHc\nikrTNHLMOWSaMsnIyyDDVLjkZZCel06mKZN0UzqZeZlkmCzb0vPSSclN4XzOeVJyU4q+qLya0c6I\nr4svvi6+NKjUgHY121HVpSq+Lr74uPhQ1aUq1V2rYzRIS80WeboYCXLxIsjv31cemcwFnEvL4Wxq\nDqdTcziTks2Z1BxOp2RzLi2HY4mZJGXkkptfcM1juzna4+lsxMPZiKezPR5OxsueX3xs2cfVwR5X\nR3tcHAy4ONjj4mjAxWjA3iDddtZQ7GBXShmAmUA3IA7YpZT6WdO0A8U9dknTNA2zZqZAK7hirWka\n+QX55BfkYyowXbmYLz3OL8i3PNcs26/Y/7L9csw55ORfWrLN2eTm5xZtz87PvnIfc84t1e9s74yb\n0Q03Bzfcje5FYV3JqZJlcayEl6NX0XMvRy/cHdylz1tck9Fgh18lF/wquVx3H03TyMozk5yRR1Jm\nLskZeSRn5JKUkUtyZh6p2SbSsvNJyzZxIjmLtBwTqdkmsvLMt1SDg70drhfD3sGAi6M9rg4GnIwG\nHO3tcLC3u2xtuOLxxdeuft1osMNgpzAaVOH68ud22Nsp7C++ZmeHwVC4LtynPH46tUaLvRVwRNO0\nYwBKqW+AhwCrB/ui6EWsObHmyjAuKLhuOF+9vUArwFxw2T6UzvcLjgZHHA2OONk74WzvjJPBCSd7\ny+Lh4GF5XLjN2d4ZJ3snHA2ORaHtZrQs7g7uuBpdi9b2dtKTJkqXUgpXR0tr29/7+v8BXM1kLiAt\n20RaTj6p2SYyc/PJyjOTlZdPZq5lbXl+aVu26dJr6Tn55OabycsvILdwsTw2YzIX59+xhoECjORj\nxIwBMwYKsKMAu8LX7FUBDgZwsNMwKg17ZdlmtNMwoGFftO3SY4MqwIiGoei55bEdGvf26E1w44bF\nqPnmrJEMNYFTlz2PA8Ku3kkpNQIYAeDv739HJ3I0OOLh4IGdssOgDJa1nWV9xTZ1adu/ttv9e79r\n/ZzBzoDRznhpMVx6bG9nf83t13p+sT4hKjKjwQ5vN0e8r770ssAMeRmQmwG56WDKBFMO5GcXrnPA\nlH3DtWbKpsCUQ0G+Cc2cS0G+Ccx5aOZ8MOeB2YQqKFzMJpSWX7g2YSgwXbvgm9GAW/sQ8i/HUhoD\nZT/Yb4mmaXOBuWC5KuZOjtG3cV/6Nu5r1bqEEMVgyoHsC1ct5y97nGIJ7IvhnZd+KcTzMsCUdfvn\ntLMHe2cwOoG9M8rohMHgiMFgBIMDODiAwdXy2M7esjYYLYud8TrP7S2P7ezBzgDKzrLYGUAZLlvb\nXfX8RvsartyuFCgD9TytN5PX9Vgj2OOBWpc99yvcJoQobzQNclIgI6FwOXdpnZlY+PwcZBWG942C\n2c4Izl7g4AaO7pbFrRp4u13aVvRa4TYHt6LAvu7aIF2QN2ONP6FdwN1KqbpYAv1JQJrVQpRFpmxI\njYfUk5Aad9lyqnAdD9e6osrOCG5Vwc0XPGpC9WBwrnTVUvnK5w6ullaqKHXFDnZN0/KVUs8Bq7Fc\n7rhQ07T9xa5MCHFnctLg/DE4fxSSC9fnj1mWzMSrdlbgXh08/aB6CDR6wPL8Yoi7+lrWzpUkpMsR\nq3ym0TRtFbDKGscSQtyijERIOAAJ/0DiP5AQYwnxq8PbvQZUrgcN7oNKtcGzVuHiBx41LH3NwqZI\nZ5UQZZ0pG85Gw7m/LSF+ccm6bKgHJy/wbQwN7oXK9cG7vmVdua6lS0RUKBLsQpQleVlwLhpOR8GZ\nKMs6MQa0wmvrHNzApxE0vB98m1jC3LexpetEukpEIQl2IfRSUABJh+DUDojbCfG7IfHgpRB3qQI1\nQiwhXiMEqgVZulDs5L4IcWMS7EKUltwMiI+EUzsvhXlO4bDHzpWgZqjly8vqIZYg96gprXBxRyTY\nhSgpOWlw4k84tsmyPhcNWuEAWj6NocnDUCvMsnjXlxAXViPBLoS1mHIsrfBjm+D4JkvXimYGeyeo\n1Qraj7eEuF8LSwtdiBIiwS7EnSowW77gvBjkJ/+yjGGiDFCzObR/Eep2tIS6vUxPJ0qPBLsQt0rT\nIOmwJcSPbYTYLZf6yH2bQIshUK8j1G4LTh66lioqNgl2IW4kNb4wyAtb5elnLNs9/aFxL6jXCep2\nsNydKUQZIcEuxOWyzkPsVkuL/PgmSD5i2e7ibQnwuh0trfJKdeXLTlFmSbCLii0vC05uv9QqP7MX\n0MDoCnXaXupe8W0q14+LckOCXVQsZpPlapWLQR630zIZg50R/FpCp0mW7pWaLWQMFVFuSbAL21ZQ\nYLl+/Phmy3Jim2WyBxRUC4SwkVC3E9RuLWOqCJshwS5si6ZZbtM/vtnSKo/dapkQAsD7Lgh87NIX\nni6V9axUiBIjwS7KN02DC8fh+BZLmMdusczwA5YrVxo+UPilZ3vLELVCVAAS7KJ8MedbulZO/mX5\n0vPkX5Bx1vKaW7XCEC9cKtXRtVQh9CLBLsq2vEyIi7gU5HG7LJMgg6VFXrcD+IdBnQ5Q5W65BFEI\nJNhFWZKfBwn7LVetnN5dOIxtTOHAWQqqBkDwU+AfbllKYbZ3IcojCXahj/w8SDpomRnoYoif/fvS\nRMrOlS2XHDZ+EPxaQa2W4OSpb81ClBMS7KJkaZrly8yz0Za+8XP7LUvSQSjIt+xjdLWMPx42Amo0\ntwyg5VVbulWEuEMS7MI6zCa4cAKSD1sGyko+DElHLAGelXxpPw8/qNrUMjdn1aaW7pUqd4OdQb/a\nhbAxEuzi1uVmQMrJy5YTcP6YJcgvHL/UAgfLtG5V7rbMCFQ1wBLivk3k2nEhSoEEu7DIy7JcNph+\nzjKCYcY5SIu/FOIXTkD2+St/xt7ZckmhbyNLX3iVu8H7bqhyl0wkIYSOihXsSqk+wBtAY6CVpmkR\n1ihKWEGB2XLHZVayZcTCrGRLMGclW5aMBEg/e2nJTf33MQyO4OUPlWpDjWaWx17+lv5vr9rgWkX6\nwYUog4rbYo8GegNzrFCLAMuXjeY8MGWBKdvS/ZGbVrikW+bRzE0vXK7ennYpxHNSAe3a5zA4gltV\ncK8GPg0soxe6VwP36oXbq1ueO1eS4BaiHCpWsGua9g+AKg//+DXNcj10gdmy1syFj82WgaKueH7Z\n2pxXuJiu8dh0ne2FjwsKn+dfFtT5OZcem7Is82RefHzxtYsTHt+M0RUc3S2z9Ti6WxYvf8ulgi7e\nlv5sF29LQLt4X9pmdJHAFsKGla8+9k3/hb+/vX4gXx7cV4f0rYalNdkZweBgGf7V6AJG58K1k+Wx\nU3XLRMdFr12+uFhec3QHR48rw9vRHRzcwVC+/vqEEKXjpsmglFoLVLvGS69omvbTrZ5IKTUCGAHg\n7+9/ywVewa2q5coKO4NlwuCitZ1lreyueu3q5xf3tbtq23V+9mIoX7G+xmO76+wjrWIhhA6Upl2n\nH/Z2DqLURmD8rX55GhoaqkVEyPesQghxO5RSkZqmhd5sP5nrSwghbEyxgl0p9YhSKg5oDaxUSq22\nTllCCCHuVHGvivkR+NFKtQghhLAC6YoRQggbI8EuhBA2RoJdCCFsjAS7EELYGAl2IYSwMVa5Qem2\nT6pUInCi1E9cfFWAJL2LKGXynm1fRXu/UH7fc21N03xutpMuwV5eKaUibuWuL1si79n2VbT3C7b/\nnqUrRgghbIwEuxBC2BgJ9tszV+8CdCDv2fZVtPcLNv6epY9dCCFsjLTYhRDCxkiw3wGl1EtKKU0p\nVUXvWkqaUup/SqkYpdQ+pdSPSikvvWsqKUqp+5RSB5VSR5RSk/Sup6QppWoppTYopQ4opfYrpcbp\nXVNpUUoZlFJ7lFK/6l1LSZBgv01KqVpAd+Ck3rWUkjVAgKZpQcAh4D8611MilFIGYCZwP9AEeEop\n1UTfqkpcPvCSpmlNgHDg2Qrwni8aB/yjdxElRYL99n0E/B9QIb6c0DTtD03T8guf/gX46VlPCWoF\nHNE07ZimaXnAN8BDOtdUojRNO6Np2u7Cx+lYgq6mvlWVPKWUH/AAMF/vWkqKBPttUEo9BMRrmrZX\n73sXuFAAAAF9SURBVFp08jTwm95FlJCawKnLnsdRAULuIqVUHaAZsEPfSkrFdCyNMx1muC8dMs39\nVW40eTfwMpZuGJtyKxOWK6VewfLRfWlp1iZKnlLKDfgeeF7TtDS96ylJSqmeQIKmaZFKqU5611NS\nJNivomnaPdfarpQKBOoCe5VSYOmS2K2UaqVp2tlSLNHqrveeL1JKDQZ6Al01270+Nh6oddlzv8Jt\nNk0pZcQS6ks1TftB73pKQVugl1KqB+AEeCilvtQ0rb/OdVmVXMd+h5RSsUCopmnlcSChW6aUug+Y\nBnTUNC1R73pKilLKHsuXw12xBPouoK+maft1LawEKUsL5XPgvKZpz+tdT2krbLGP1zStp961WJv0\nsYub+RRwB9YopaKUUrP1LqgkFH5B/BywGsuXiMttOdQLtQUGAF0K/26jCluyopyTFrsQQtgYabEL\nIYSNkWAXQggbI8EuhBA2RoJdCCFsjAS7EELYGAl2IYSwMRLsQghhYyTYhRDCxvw/IL1meIhxdlUA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcbbcc8b160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def loss(f,y):\n",
    "    return -nd.log(nd.LogisticRegressionOutput(f*y, nd.ones_like(f)))\n",
    "\n",
    "# getting data ready\n",
    "thelabel = nd.ones_like(output)\n",
    "with ag.record():    # start recording\n",
    "    theloss = loss(output, thelabel)\n",
    "theloss.backward()   # and compute the gradient\n",
    "\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='Logistic loss')\n",
    "plt.plot(output.asnumpy(), output.grad.asnumpy(), label='Gradient of logistic loss')\n",
    "# now compute the loss for y=-1\n",
    "theloss = loss(output, -thelabel)\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='Logistic loss for y=-1')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the loss function asymptotes the negative identity function for very negative values of $yf$. For very large positive values of $yf$, the loss converges to 0. This is desirable, since large values of $yf$ mean that the classification is correct. \n",
    "\n",
    "## Classification - Multiclass Softmax\n",
    "\n",
    "One way of dealing with multiple classes is to turn it into $n$ binary classification problems. That is, we simply test: 'is it class 1', 'is it class 2', ... 'is it class n'. In theory this sounds like a splendid idea. After all, this should be just as easy as determining which class it is. Unfortunately, that's not quite the case. Imagine the situation where none of the $n$ classifiers wants to take responsibility. Or imagine the case where more than one claims that it's his turn. Obviously there has to be a better way. Indeed, there is.\n",
    "\n",
    "If we have a vector $f \\in \\mathbb{R}^n$ of scores, where the coordinate, say $f_i$ is large whenever we think that the correct class is $i$, then we can map $f$ into a probability vector via\n",
    "\n",
    "$$p(y=i|f) \\propto \\exp(f_i) \\text{ and hence } p(y=i|f) = \\frac{\\exp(f_i)}{\\sum_j \\exp(f_j)}$$\n",
    "\n",
    "Here the normalization by $\\sum_j \\exp(f_j)$ is needed such that all the terms sum up to 1. Consequently the negative log-likelihood $-\\log p(y|f)$, i.e. the quantity that we would want to minimize in this case is given by \n",
    "\n",
    "$$-\\log p(y=i|f) = \\log \\left[\\sum_{j} \\exp(f_j)\\right] - f_i$$\n",
    "\n",
    "In ``gluon`` the relevant function is [mxnet.gluon.loss.SoftmaxCrossEntropyLoss](http://mxnet.io/api/python/gluon.html#mxnet.gluon.loss.SoftmaxCrossEntropyLoss). Let's check that this is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Softmax loss is 1.5590695142745972.\n",
      "Class 4 has negative log-likelihood 1.5590696334838867.\n"
     ]
    }
   ],
   "source": [
    "loss = gluon.loss.SoftmaxCrossEntropyLoss()\n",
    "\n",
    "f = nd.random_normal(shape=(1,10))\n",
    "y = nd.array([4]) #class 4 is true\n",
    "\n",
    "print('Softmax loss is {}.'.format(loss(f,y).asscalar()))\n",
    "\n",
    "# now compute this by hand\n",
    "\n",
    "p = nd.exp(f)\n",
    "p = p / nd.sum(p)\n",
    "print('Class 4 has negative log-likelihood {}.'.format(-nd.log(p[0,4]).asscalar()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The softmax loss has a rather nice property that is worth pointing out: its gradient is given by the difference between the conditional class probabilities $p(y=i|f)$ and the indicator vector $e_j$. This can be derived via \n",
    "\n",
    "$$\\partial_{f_i} \\log \\sum_j \\exp(f_j) = \\frac{\\exp(f_i)}{\\sum_j \\exp(f_j)} = p(y=i|f)$$\n",
    "\n",
    "Such a result seems to be too good to be true by chance. In fact, it holds for *every* member of a larger family of distributions, called the [Exponential Family](https://en.wikipedia.org/wiki/Exponential_family). More specifically, the derivative of the associated normalization is the expected value of the associated embedding. \n",
    "\n",
    "## Classification - Soft Margin Loss\n",
    "\n",
    "Note that the logistic loss isn't the only loss that one might encounter. For instance, in Support Vector Machines we have a soft-margin loss. It is $0$ whenever data is correctly classified with some confidence, say $y f(x) > 1$. Otherwise we impose a linear penalty. In math this amounts to\n",
    "\n",
    "$$l(y,f) = \\mathrm{max}(0, 1- yf)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYlXX+//Hn57DjgiBo7rgrCIIgiDq5r9MPM9dScQfN\nnBqnUpvyqzaWleVSKioqaGYujTVZ6ojllriAg6m4LxmmiKa4AMry+f1heWWZGzfccM77cV1eCtzn\nc7+OHl/c3Oc+76O01gghhLAeFrMDCCGEMJYUuxBCWBkpdiGEsDJS7EIIYWWk2IUQwspIsQshhJWR\nYhdCCCsjxS6EEFZGil0IIayMvRk79fT01N7e3mbsWgghSqykpKSLWmuvB21nSrF7e3uTmJhoxq6F\nEKLEUkr98DDbyakYIYSwMlLsQghhZaTYhRDCyphyjl0UDzk5OaSmppKdnW12FFGMOTs7U7VqVRwc\nHMyOIh6SFLsNS01NpUyZMnh7e6OUMjuOKIa01ly6dInU1FRq1qxpdhzxkAw5FaOUKqeUWq2UOqyU\nOqSUCjNiXVG4srOzKV++vJS6+FNKKcqXLy8/1ZUwRh2xzwTWa617KqUcAVeD1hWFTEpdPIg8Rkqe\nAh+xK6XcgCeBhQBa61ta6ysFXfde/nfmMvO2nEDezk8IUdJordmWuq1I+suIUzE1gXRgsVLqf0qp\nGKVUqd9vpJSKVEolKqUS09PTH2tH/957lrfXHeatrw9JuVuJKVOm4Ovri7+/PwEBAezateu+22/b\ntg1fX18CAgI4dOgQn3zySREl/XNdu3blypVCOZYRVkJrzfuJ7/P8pufZcHpDoe/PiGK3B5oAc7XW\ngcANYNzvN9Jaz9daB2utg728HviK2HuaFO7LwLAaLNh2irGffU9uXn6BggtzJSQksHbtWvbu3cv3\n339PfHw81apVu+9tli1bxvjx40lOTiYtLa1Iij03N/e+X//6668pV65coecQJVNufi4TdkwgLiWO\nZxs8S0fvjoW+TyOKPRVI1Vr/eqi1mttFbziLRTEx3Je/tavLysRURn2yl+ycvMLYlSgC586dw9PT\nEycnJwA8PT2pXLkyAJs2bSIwMBA/Pz+GDBnCzZs3iYmJYeXKlbzxxhv069ePcePGsW3bNgICApg+\nffpda2/evJlWrVrRrVs3atWqxbhx41i2bBkhISH4+flx4sQJAL788ktCQ0MJDAykffv2pKWlATBx\n4kQGDBhAixYtGDBgAJmZmfTu3RsfHx+6d+9OaGjonbEY3t7eXLx4kdOnT9OwYUOGDx+Or68vHTt2\nJCsrq6j+OkUxdDPvJv/Y/A8+P/45IxuPZHzIeCyq8F8+VOAnT7XW55VSPyql6mutjwDtgJSCR7s3\npRRjOtTD3dWBSV+mMCR2D/MjgintJFduFsSkLw+S8tNVQ9f0qVyW//t/vn/69Y4dOzJ58mTq1atH\n+/bt6dOnD61atSI7O5tBgwaxadMm6tWrR0REBHPnzuWll15i+/btPPXUU/Ts2ZPNmzczbdo01q5d\ne8/19+3bx6FDh/Dw8KBWrVoMGzaM3bt3M3PmTD788ENmzJhBy5Yt2blzJ0opYmJiePfdd3n//fcB\nSElJYfv27bi4uDBt2jTc3d1JSUnhwIEDBAQE3HOfx44dY/ny5SxYsIDevXvz2Wef0b9//4L/ZYoS\n50bODV785kV2nd/FuJBx9GvYr8j2bdS3jtHAMqXU90AA8JZB6/6pwS1q8kHvxuw69TP9Fuzk5xu3\nCnuXwmClS5cmKSmJ+fPn4+XlRZ8+fYiNjeXIkSPUrFmTevXqATBw4EC2bt36yOs3bdqUSpUq4eTk\nRO3atenY8faPwH5+fpw+fRq4fS1/p06d8PPz47333uPgwYN3bh8eHo6LiwsA27dvp2/fvgA0atQI\nf3//e+6zZs2ad0o/KCjozn6EbbmcfZlhG4aRmJbIWy3fKtJSB4Mud9RaJwPBRqz1KJ5pUpWyzg48\n/8lees9LYOnQECq5uRR1DKtwvyPrwmRnZ0fr1q1p3bo1fn5+xMXFERgYaMjav57iAbBYLHc+tlgs\nd86bjx49mjFjxhAeHs7mzZuZOHHinduUKvWHawAeaZ92dnZyKsYGnb9xnqiNUaReS2VGmxm0rta6\nyDOU+Fkx7X0qsmRICOczsuk5N4GT6dfNjiQe0pEjRzh27Nidj5OTk6lRowb169fn9OnTHD9+HICl\nS5fSqlWrP9y+TJkyXLt2rUAZMjIyqFKlCgBxcXF/ul2LFi1YuXIlcPsUzf79+wu0X2GdTmecJmJd\nBGmZaUR3iDal1MEKih2gWa3yfBrZjOycPHpFJ3DgbIbZkcRDuH79OgMHDsTHxwd/f39SUlKYOHEi\nzs7OLF68mF69euHn54fFYmHEiBF/uL2/vz92dnY0btz4D0+ePqyJEyfSq1cvgoKC8PT0/NPtnn/+\nedLT0/Hx8eH111/H19cXNze3x9qnsE4pl1IYuH4gN/NusqjTIpo+0dS0LMqM68GDg4N1YbzRxon0\n60Qs3M3VrBxiBgYTWqu84fuwJocOHaJhw4ZmxygR8vLyyMnJwdnZmRMnTtC+fXuOHDmCo6Oj2dGK\nhDxW7i/xfCKjvxlNGccyzO8wH28370LZj1IqSWv9wNPeVnHE/qvaXqVZNSKMCmWdiFi0m02H0syO\nJKxEZmYmLVu2pHHjxnTv3p05c+bYTKmL+9vy4xZGxI/Ay9WLJV2WFFqpPwqru0awcjkXVo1ozqDF\nu4lcmsT7vRrzdGAVs2OJEq5MmTLydo7iD9aeXMvr21+ngUcD5rafi7uzu9mRACs7Yv+VRylHPhne\njBBvD15akUzsd6fMjiSEsDLLDi1j/LbxBFUMYmGnhcWm1MFKix2gtJM9iwc3pYNPRSZ+mcKM+KMy\nX0YIUWBaa+Ykz2Hq7qm0rdaWOe3nUMrh0S+NLUxWW+wAzg52zO3XhJ5BVZkRf4xJX6aQny/lLoR4\nPPk6n6m7pzJ331y61e7G+63fx8nO6cE3LGJWd4799+ztLLzbw59yLg7EbD9FRlYO7/b0x8HOqr+n\nCSEMlpOfw4TvJrD25FoifCL4R/A/imTuy+MonqkMZrEo/vnXhrzSqT5r/neWEUuTZHhYMZKWlsZz\nzz1HrVq1CAoKIiwsjDVr1jz2ehMnTmTatGkATJgwgfj4+MdaJzk5ma+//vqxcwjrkZ2bzd+//Ttr\nT67lb4F/4+Xgl4ttqYONFDvcHh42qk0d3ny6Ed8cuUDEot1czc4xO5bN01rz9NNP8+STT3Ly5EmS\nkpL49NNPSU1NvWu7B43O/TOTJ0+mffv2j3VbKXYBcO3WNaI2RrE1dSuvh77OcP/hxf5dpWym2H81\noFkNZvYNZO8Pl3l2/k4uXr9pdiSb9s033+Do6HjXK0tr1KjB6NGjiY2NJTw8nLZt29KuXTuuX79O\nu3btaNKkCX5+fnzxxRd3bjNlyhTq1atHy5YtOXLkyJ3PDxo0iNWrVwOQlJREq1atCAoKolOnTpw7\ndw6A1q1bM3bsWEJCQqhXrx7btm3j1q1bTJgwgRUrVhAQEMCKFSuK6G9EFCcXsy4yZMMQvk//nnee\nfIc+DfqYHemhWP059nsJb1yZss72jPg4iV7Rt4eHVXW38bdpXTcOzhs8/+QJP+gy9b6bHDx4kCZN\n/nx8/69vwuHh4UFubi5r1qyhbNmyXLx4kWbNmhEeHs7evXv59NNPSU5OJjc3lyZNmhAUFHTXOjk5\nOYwePZovvvgCLy8vVqxYwT//+U8WLVoE3P6JYPfu3Xz99ddMmjSJ+Ph4Jk+eTGJiIh999FHB/y5E\nifPT9Z+I3BhJ2o00Pmz3IS2rtDQ70kOzyWIHaF2/Ah8PDWVI7B56zr1d7nUrljE7ls0bNWoU27dv\nx9HRkVGjRtGhQwc8PDyA26dtXnvtNbZu3YrFYuHs2bOkpaWxbds2unfvjqvr7W/O4eHhf1j3yJEj\nHDhwgA4dOgC3RwRUqlTpztefeeYZQEbtittOXDlB5MZIsnKzmN9xPoEVjJk4WlRsttgBgr09WBEV\nxoCFu+k9L4HYwSE0rmajb3H2gCPrwuLr68tnn3125+PZs2dz8eJFgoNvj8P47ejcZcuWkZ6eTlJS\nEg4ODnh7e5Odnf1Q+9Fa4+vrS0JCwj2//uu4XTs7u8c+ny+sw4GLBxgZPxI7ZcfiToup71Hf7EiP\nzObOsf9ew0pl+WxkGKWd7XluwU52HL9odiSb0rZtW7Kzs5k7d+6dz2VmZt5z24yMDCpUqICDgwPf\nfvstP/zwAwBPPvkkn3/+OVlZWVy7do0vv/zyD7etX78+6enpd4o9JyfnrjfVuBcjxgKLkmXnuZ0M\n3TCUUg6lWNplaYksdZBiB6BG+VKsHtGcKu4uDFq8hw0Hz5sdyWYopfj888/ZsmULNWvWJCQkhIED\nB/LOO+/8Ydt+/fqRmJiIn58fS5YsoUGDBgA0adKEPn360LhxY7p06ULTpn8cl+ro6Mjq1asZO3Ys\njRs3JiAggB07dtw3W5s2bUhJSZEnT23Eph828Xz881QuXZklXZZQrez931i9OLOqsb0FdSXzFoNj\n97DvxytM7eFP7+CS+w/7MGQUq3hY1v5YWXNsDRMTJtLIsxFz2s3Bzal4ztq3ybG9BVXO1ZFlw0Jp\nUceTV1d/z4KtJ82OJIQoZHEH45iwYwLNKjVjQYcFxbbUH4Uhxa6UOq2U2q+USlZKFb9D8Ufg6mhP\nzMBg/upXiSlfH+K9DYdleJgQVkhrzay9s5iWOI2ONTryYdsPcXWwjsuejbwqpo3W2iqeeXSyt2PW\ns4GUdbFn9rcnuJyZw5vdGmFnKd6vNhNCPJy8/Dym7JrCqqOr6FG3B280ewM7i53ZsQxj05c73o+d\nRfFWdz/KuToyd/MJrmbl8EHvABzt5eyVECVZTl4O47ePZ8PpDQxtNJQXm7xY7EcEPCqjil0D/1VK\naWCe1nr+7zdQSkUCkQDVq1c3aLeFSynF2M4NKOfiwNvrDnM1O5fo/k1wdZTvh0KURJk5mYzZPIbv\nfvqOMUFjGNxosNmRCoVRh58ttdZNgC7AKKXUk7/fQGs9X2sdrLUO9vLyMmi3RSOqVW3e6eHH9mPp\n9I/ZRUamDA8ToqTJuJlB5MZIEs4lMKn5JKstdTCo2LXWZ3/5/QKwBggxYt3ipE/T6szp14QDZ6/S\nZ34CF64+3Csexf1NmTIFX19f/P39CQgIYNeuXffdftu2bfj6+hIQEMChQ4f45JNPiijpn+vatStX\nrlx56O3T09MJDQ0lMDCQbdu2FWKyB/voo4+oU6cOSikuXrSKp8juKT0zncEbBpNyKYX3W73PM3Wf\nMTtSoSpwsSulSimlyvz6Z6AjcKCg6xZHnRtVYtGgppz5OZOe0QmcuXTvV0iKh5OQkMDatWvvDPqK\nj4+nWrX7v3Zg2bJljB8/nuTkZNLS0oqk2B80YuDrr7+mXLmHH0WxadMm/Pz8+N///sdf/vKXh7pN\nXl7hvH9AixYtiI+Pp0aNGoWyfnHw47UfiVgXQeq1VGa3m037Go83xrkkMeKIvSKwXSm1D9gNfKW1\nXm/AusVSy7qeLBsWytXsHHpG7+Dw+atmRyqxzp07h6en5505LZ6enlSuXBm4XX6BgYH4+fkxZMgQ\nbt68SUxMDCtXruSNN96gX79+jBs3jm3bthEQEMD06dPvWnvz5s20atWKbt26UatWLcaNG8eyZcsI\nCQnBz8+PEydOAPDll1/eOXpu3749aWlpwO036xgwYAAtWrRgwIABZGZm0rt3b3x8fOjevTuhoaH8\n+iI7b29vLl68yOnTp2nYsCHDhw/H19eXjh07kpWVdVeu5ORkXn31Vb744gsCAgLIyspi+fLl+Pn5\n0ahRI8aOHXtn29KlS/OPf/yDxo0b3zXj5sSJE3dNxDx27Nh9J2TeT2BgIN7e3o9125Lg6OWjRKyL\n4FrONWI6xhBWOczsSEWiwM8Caq1PAo0NyFJiBFZ3Z2VUGAMW7qJ3dAKLB4cQVKP4vEP543hn9zsc\n/vmwoWs28GjA2JCxf/r1jh07MnnyZOrVq0f79u3p06cPrVq1Ijs7m0GDBrFp0ybq1atHREQEc+fO\n5aWXXmL79u089dRT9OzZk82bNzNt2jTWrl17z/X37dvHoUOH8PDwoFatWgwbNozdu3czc+ZMPvzw\nQ2bMmEHLli3ZuXMnSiliYmJ49913ef/99wFISUlh+/btuLi4MG3aNNzd3UlJSeHAgQMEBATcc5/H\njh1j+fLlLFiwgN69e/PZZ5/Rv3//O18PCAi4axzwTz/9xNixY0lKSsLd3Z2OHTvy+eef8/TTT3Pj\nxg1CQ0Pv5PlV7dq1cXNzIzk5mYCAABYvXszgwbfPF//973/n22+//UOuvn37Mm7cuPv/g1mZ5AvJ\nPL/peVzsXIjtFEsd9zpmRyoycu3eY6pXsQyrRzTHo5Qj/WN2seVoutmRSpzSpUuTlJTE/Pnz8fLy\nok+fPsTGxnLkyBFq1qxJvXr1ABg4cCBbt2595PWbNm1KpUqVcHJyonbt2nTs2BEAPz+/O6N5U1NT\n6dSpE35+frz33nt3DQYLDw/HxcUFgO3bt9O3b18AGjVqhL+//z33WbNmzTul/zAjgPfs2UPr1q3x\n8vLC3t6efv363bmvdnZ29OjR4563GzZsGIsXLyYvL48VK1bw3HPPATB9+nSSk5P/8MvWSv27s98R\nuTESdyd3lnRdYlOlDnIde4FU83Bl1YjmDFy0m2Fxe5jeJ4Cn/CubHeux3O/IujDZ2dnRunVrWrdu\njZ+fH3FxcQQGGjP7+tdTPAAWi+XOxxaL5c5589GjRzNmzBjCw8PZvHkzEydOvHOb344Mfpx92tnZ\n/eFUzKNwdnbGzu7eL5rp0aMHkyZNom3btgQFBVG+fHngwUfsnTp1Ii0tjeDgYGJiYh47W3G2/vR6\nxm8bT51ydZjbfi6eLp5mRypyUuwF5FXGieWRzRgWt4fRy//H1axcngstGdfpm+3IkSNYLBbq1q0L\n3D7/XKNGDerXr8/p06c5fvw4derUYenSpbRq1eoPtzdirG5GRgZVqlQBIC4u7k+3a9GiBStXrrwz\n8XH/fmPebSokJIS//e1vXLx4EXd3d5YvX87o0aMfeDtnZ2c6derEyJEjWbhw4Z3P//65ht/bsGFD\ngTMXZ6uOruLNhDcJrBDIh+0+pKxjWbMjmUJOxRjAzcWBJUNCaV3Pi9fW7GfO5uMyX+YhXL9+nYED\nB+Lj44O/vz8pKSlMnDgRZ2dnFi9eTK9evfDz88Nisdz1nqi/8vf3x87OjsaNGz+w0P7MxIkT6dWr\nF0FBQXh6/vmR3fPPP096ejo+Pj68/vrr+Pr64uZW8GFRlSpVYurUqbRp04bGjRsTFBREt27dHuq2\n/fr1w2Kx3DnF9DhmzZpF1apVSU1Nxd/fn2HDhj32WmbSWhOzP4bJCZNpWaUl0R2ibbbUQcb2Gion\nL5+XV+3ji+SfiHyyFuO7NCjWL1W29lGsRsrLyyMnJwdnZ2dOnDhB+/btOXLkCI6OjqZlmjZtGhkZ\nGbz55puFvq/i/FjRWvN+4vvEpcTRtWZX/tXyXzhYHMyOVSgedmyvnIoxkIOdhem9Ayjn4sD8rSe5\nknmLt7r7YW8nPxiVdJmZmbRp04acnBy01syZM8fUUu/evTsnTpzgm2++MS1DcZCbn8vkhMmsOb6G\nZxs8y7iQcViU/H+TYjeYxaKYGO6Lm6sjszYdIyMrh5l9A3F2sJ7JcbaoTJkyFKefMtesWWN2BNPd\nzLvJ2K1j2XRmEyMaj+D5xs8X65+Qi5J8aysESinGdKjHhKd82HAwjSGxe7h+s3i+QbI8FyAepDg+\nRm7k3GBU/Cg2ndnE2KZjGRUwSkr9N6TYC9GQljV5v1djdp36mX4LdnL5xi2zI93F2dmZS5cuFcv/\nuKJ40Fpz6dIlnJ2dzY5yx+XsywzbMIzEtESmtJxCf5/+D76RjZFTMYWsR1BVyro4MOqTvfSal8DS\noSFUcnMxOxbAnash0tPlxVXizzk7O1O1alWzYwBw/sZ5ojZGkXotlemtp9OmehuzIxVLclVMEdl5\n8hLD4hJxc3Fg6dAQanmVNjuSECXK6YzTRG6M5Oqtq3zY9kOaPtHU7EhFTt7MuphpVqs8n0Y2Izsn\nj17RCRw4m2F2JCFKjEOXDjFw/UBu5t1kUadFNlnqj0KKvQg1quLGyhFhONlbeHb+Tnaf+tnsSEIU\ne0lpSQzZMARHO0diO8fiU97H7EjFnhR7EavtVZrVI5tToawTAxbu4pvDaWZHEqLY2vLjFqI2RuHl\n6sXSLkup6VbT7EglghS7CSqXc2FlVBj1nyjD8CVJfP6/s2ZHEqLYWXtyLS9++yJ1ytUhtnMsT5R6\nwuxIJYYUu0nKl3bik+HNCPH24KUVycR+d8rsSEIUG8sOLWP8tvEEVQxiYaeFeDh7mB2pRJFiN1Fp\nJ3sWD25KB5+KTPwyhRnxR+WacmHTtNbMTZ7L1N1TaVOtDXPaz6GUw6OPT7Z1Uuwmc3awY26/JvQM\nqsqM+GNM+jKF/Hwpd2F78nU+7+x5hzn75tCtdjc+aP0BTnZOD76h+AN5gVIxYG9n4d0e/ri5OLBw\n+ykysnJ4t6c/DjI8TNiInPwcJnw3gbUn1zLAZwAvB78sw7wKwLBiV0rZAYnAWa31U0atayssFsXr\nf22Iu6sD0/57lGvZOXz0XBMZHiasXnZuNi9veZktqVsYHTia4X7DZe5LARn5LfFF4JCB69kcpRQv\ntK3Lm083YtPhC0Qs2s3V7ByzYwlRaK7dusaI+BFsTd3K66GvE+kfKaVuAEOKXSlVFfgrYJ1voljE\nBjSrwcy+gez94TLPzt/Jxes3zY4khOEuZV1i6Iah7Luwj3eefIc+DfqYHclqGHXEPgN4Fcg3aD2b\nF964MgsGBnMi/Tq9oxNIvZxpdiQhDPPT9Z8YtH4QpzJOMavtLLrU7GJ2JKtS4GJXSj0FXNBaJz1g\nu0ilVKJSKlGmCT6cNvUr8PHQUNKv36RXdALHLxTsjZuFKA5OXjlJxLoILmVdYn7H+fyl6l/MjmR1\njDhibwGEK6VOA58CbZVSH/9+I631fK11sNY62MvLy4Dd2oZgbw9WRoWRk6fpFZ3Avh+vmB1JiMd2\n4OIBBq4fSG5+Los7LyawQqDZkaxSgYtdaz1ea11Va+0N9AW+0VrL5HsDNaxUls9GhlHa2Z7nFuxk\nx/GLZkcS4pHtOreLoRuGUsqhFEu6LKG+R32zI1ktuVC0hKhRvhSrRzSnirsLgxbvYcPB82ZHEuKh\nbTqziZHxI6lcujJLuiyhetnqZkeyaoYWu9Z6s1zDXngqlnVmZVQYvlXKMvLjJFYm/mh2JCEeaM2x\nNYzZPIaG5RsS2zmWCq4VzI5k9eSIvYQp5+rIx0NDaVHHk1dXf0/MtpNmRxLiT8UdjGPCjgmEPhHK\ngg4LcHNyMzuSTZBiL4FKOdkTMzCYv/pV4l9fHWLahiMyPEwUK1prZu2dxbTEaXSo0YGP2n2Eq4Or\n2bFshsyKKaGc7O2Y9WwgZV3s+ejb41zOvMXkbo2ws8ir9oS58vLzmLJrCquOrqJH3R680ewN7Cwy\nGqMoSbGXYHYWxVvd/Sjn6sjczSfIyMrhg94BONrLD2LCHDl5Oby2/TXWn17P0EZDebHJizIiwARS\n7CWcUoqxnRtQzsWBt9cd5lp2LnP7N8HVUf5pRdHKzMlkzJYxfHf2O8YEjWFwo8FmR7JZcmhnJaJa\n1WbqM35sO5bOgIW7yciU4WGi6GTczCBqYxQJPyUwMWyilLrJpNitSN+Q6sx+rgn7UzPoMz+BC1ez\nzY4kbEB6ZjqDNwzm4KWDTGs1jR71epgdyeZJsVuZLn6VWDSoKWd+zqRndAJnLsnwMFF4frz2IxHr\nIki9lsrsdrPpUKOD2ZEEUuxWqWVdT5YNC+Vqdg49o3dw+PxVsyMJK3T08lEGrhvItZxrxHSMIaxy\nmNmRxC+k2K1UYHV3VkaFoRT0jk4g6YfLZkcSViT5QjKD1g9CoYjtFIu/l7/ZkcRvSLFbsXoVy7B6\nRHM8SjnSP2YXW4/KuGRRcDvO7iByYyTuTu4s6bqEOu51zI4kfkeK3cpV83Bl1YjmeHuWYmjcHr76\n/pzZkUQJtv70ekZ9M4rqZaoT1yWOKqWrmB1J3IMUuw3wKuPEp5HNCKhWjheW7+WTXWfMjiRKoFVH\nV/Hqllfx9/RnUedFeLp4mh1J/Akpdhvh5uLAkiGhtK7nxWtr9jNn83GZLyMeitaamP0xTE6YTMsq\nLYnuEE1Zx7JmxxL3IcVuQ1wc7ZgfEUy3gMq8u/4Ib687LOUu7ktrzQdJHzBz70y61uzKzLYzcbF3\nMTuWeAB53bmNcbCzML13AG4uDszfepIrmbd4q7sf9nbyPV7cLTc/l8kJk1lzfA196/dlfOh4LEoe\nJyWBFLsNslgUk8J9KefqyKxNx7ialcvMZwNwspcJfOK2m3k3Gbd1HPFn4onyj2JUwCgZ5lWCyLdf\nG6WUYkyHekx4yof1B88zJHYP12/mmh1LFAM3cm4wKn4U8WfiGdt0LC8EviClXsJIsdu4IS1r8n6v\nxuw8+TP9Fuzk8o1bZkcSJrqSfYVhG4aRmJbIlJZT6O8j70tfEhW42JVSzkqp3UqpfUqpg0qpSUYE\nE0WnR1BVovsHcej8NXrNS+BcRpbZkYQJzt84z8D1Azl6+SjTW08nvHa42ZHEYzLiiP0m0FZr3RgI\nADorpZoZsK4oQh18KhI3OITzGdn0nJvAqYs3zI4kitAPV39g4LqBpGWmEd0hmjbV25gdSRRAgYtd\n33b9lw8dfvkl19CVQGG1y7N8eDOycvLoFb2Dgz9lmB1JFIFDlw4RsS6CrNwsFnZaSNMnmpodSRSQ\nIefYlVJ2Sqlk4AKwUWu9y4h1RdHzq+rGqhFhONpZ6DtvJ7tP/Wx2JFGIktKSGLJhCI52jsR2icW3\nvK/ZkYQBDCl2rXWe1joAqAqEKKUa/X4bpVSkUipRKZWYni7DqIqz2l6lWT2yORXKOjFg4S6+OZxm\ndiRRCLZiylXwAAAW+ElEQVSmbiVqYxSeLp4s6byEWm61zI4kDGLoVTFa6yvAt0Dne3xtvtY6WGsd\n7OXlZeRuRSGoXM6FlVFh1KtYhsglSXz+v7NmRxIGWntyLS9+8yK1y9UmrksclUpXMjuSMJARV8V4\nKaXK/fJnF6ADcLig6wrzlS/txCfDQwn2duelFcnE7ThtdiRhgE8OfcL4beMJrBjIwo4L8XD2MDuS\nMJgRR+yVgG+VUt8De7h9jn2tAeuKYqCMswOxg0Po4FOR//vPQWbGH5P5MiWU1pq5++by9u63aV2t\nNXPbz6W0Y2mzY4lCUOCRAlrr74FAA7KIYsrZwY65/Zow7t/7mR5/lMuZt5jwlA8Wi7wasaTI1/m8\nu+ddlh1aRnjtcCY1n4S9RSaKWCv5lxUPxd7Owrs9/HFzcWDh9lNkZOXwbk9/HGR4WLGXk5/DhO8m\nsPbkWvo37M8rTV+RYV5WTopdPDSLRfH6Xxvi7urAtP8e5Vp2Dh891wRnBxkeVlxl52bzypZX2Jy6\nmdGBoxnuN1zmvtgA+bYtHolSihfa1uXNpxux6fAFIhbt5mp2jtmxxD1cu3WNEfEj2JK6hddDXyfS\nP1JK3UZIsYvHMqBZDWb0CWDvD5d5dv5OLl6/aXYk8RuXsi4xdMNQ9l3Yx9S/TKVPgz5mRxJFSIpd\nPLZuAVVYMDCYE+nX6R2dQOrlTLMjCeCn6z8xaP0gTmWcYmbbmXSt1dXsSKKISbGLAmlTvwIfDw0l\n/fpNekUncPzCNbMj2bSTV04SsS6CS1mXmNdhHk9WfdLsSMIEUuyiwIK9PVgRGUZOnqZXdALfp14x\nO5JNOnjxIAPXDyQ3P5fFnRfTpGITsyMJk0ixC0P4VC7L6hFhlHKy59n5O9lx4qLZkWzK7nO7GbJh\nCKUcSrGkyxLqe9Q3O5IwkRS7MIy3Zyk+G9mcKu4uDFq8hw0Hz5sdySZsOrOJkfEjqVy6MnGd46he\ntrrZkYTJpNiFoSqWdWZlVBi+lcsy8uMkViX+aHYkq/b58c8Zs3kMDTwaENs5loqlKpodSRQDUuzC\ncOVcHfl4aCgt6njyyurvidl20uxIVinuYBxvfPcGoU+EsqDjAtyc3MyOJIoJKXZRKEo52RMzMJiu\nfk/wr68OMW3DERkeZhCtNbP2zmJa4jQ61OjAR+0+wtXB1exYohiRkQKi0DjZ2/Hhs00o67yfj749\nzuXMW0zu1gg7GR722PLy83hr11usPLqSHnV78EazN7CzyEgHcTcpdlGo7CyKt5/xo5yrI9FbTpCR\nlcMHvQNwtJcfFh9VTl4Or21/jfWn1zOk0RBeavKSjAgQ9yTFLgqdUopxXRrg7urA2+sOcy07l+j+\nQbg4ypHmw8rMyWTMljF8d/Y7/h70d4Y0GmJ2JFGMyWGTKDJRrWoz9Rk/th1Lp//CXWRkyvCwh5Fx\nM4OojVEk/JTAxLCJUurigaTYRZHqG1Kd2c81YX9qBn3mJ3DharbZkYq19Mx0Bm8YzMFLB5nWaho9\n6vUwO5IoAaTYRZHr4leJRYOacubnTHpGJ3DmkgwPu5cfr/1IxLoIUq+lMrvdbDrU6GB2JFFCSLEL\nU7Ss68myYaFczc6hZ/QOjpyX4WG/dezyMQauG8jVW1eJ6RhDWOUwsyOJEkSKXZgmsLo7K6PCUAp6\nz0tg75nLZkcqFpIvJDNo/SAUirjOcfh7+ZsdSZQwBS52pVQ1pdS3SqkUpdRBpdSLRgQTtqFexTKs\nHtEcd1cH+i3Yxdaj6WZHMtWOszuI3BhJOadyxHWJo457HbMjiRLIiCP2XOAfWmsfoBkwSinlY8C6\nwkZU83Bl1YjmeHuWYmjcHr76/pzZkUyx4fQGRn0ziuplqhPXJY6qZaqaHUmUUAUudq31Oa313l/+\nfA04BFQp6LrCtniVceLTyGY0rlqOF5bvZfnuM2ZHKlKrj67mlS2v4Ofpx6LOi/B08TQ7kijBDD3H\nrpTyBgKBXff4WqRSKlEplZiebts/bot7c3NxYOnQUFrX82L8v/czd/MJsyMViYX7FzIpYRItqrRg\nXod5lHUsa3YkUcIZVuxKqdLAZ8BLWuurv/+61nq+1jpYax3s5eVl1G6FlXFxtGN+RDDdAirzzvrD\nvP31IasdHqa15oPED5ixdwZdanZhVptZuNi7mB1LWAFDRgoopRy4XerLtNb/NmJNYbsc7CxM7x2A\nm4sD87ae5EpmDlO6N8Leznou4srNz2VywmTWHF9Dn/p9eC30NSzKeu6fMFeBi13dnkK0EDiktf6g\n4JGEAItFMSncl3IuDsz65jgZWTnMfDYAJ/uSP1/mVt4txm4dS/yZeKL8oxgVMEqGeQlDGXGI0AIY\nALRVSiX/8qurAesKG6eUYkzH+rzxlA/rD55nSOwert/MNTtWgdzIucHzm54n/kw8rzZ9lRcCX5BS\nF4Yr8BG71no7II9MUWiGtqxJORcHXv3se/rF7CJ2UFPcSzmaHeuRXcm+wvObniflUgpTWk4hvHa4\n2ZGElZKTeqJE6BFUlej+QRw6d5Xe8xI4n1Gyhoel3Uhj0PpBHPn5CNNbT5dSF4VKil2UGB18KhI3\nOIRzGdn0mLuDUxdvmB3pofxw9Qci1kVwPvM80R2iaVO9jdmRhJWTYhclSljt8iwf3oysnDx6Re/g\n4E8ZZke6r8M/HyZiXQRZuVks7LSQpk80NTuSsAFS7KLE8avqxsqoMBztLPSdt5Pdp342O9I9JaUl\nMXj9YBztHIntEotveV+zIwkbIcUuSqQ6FUqzamRzvMo6MWDhLr45nGZ2pLtsTd1K1MYoPF08WdJ5\nCbXcapkdSdgQKXZRYlUp58KqqDDqVSxD5JIkvkg+a3YkAL46+RUvfvMitdxqEdcljkqlK5kdSdgY\nKXZRopUv7cQnw0MJ9nbnpRXJLEk4bWqe5YeXM37beAIqBLCo0yI8nD1MzSNskxS7KPHKODsQOziE\ndg0qMuGLg8yMP1bk82W01szdN5e3dr1Fq2qtiO4QTWnH0kWaQYhfSbELq+DsYEd0/yb0aFKV6fFH\nmfRlCvn5RVPu+Tqfd/a8w5zkOYTXDmd66+k42TkVyb6FuBdDhoAJURzY21l4r6c/bi4OLPruFFez\ncninpz8OhTg8LCc/h//77v/48uSX9G/Yn1eaviLDvITppNiFVbFYFG881RCPUg5M++9Rrmbn8NFz\nTXB2MH54WHZuNq9seYXNqZt5IeAFIv0jZe6LKBbk0EJYHaUUL7Sty5tPN2LT4QsMXLSbq9k5hu7j\n2q1rjIwfyZbULfwz9J9ENY6SUhfFhhS7sFoDmtVgRp8Akn64zLPzd3Lx+k1D1r2UdYmhG4aSfCGZ\nqX+ZSt8GfQ1ZVwijSLELq9YtoAoLIoI5kX6d3tEJnL2SVaD1zl0/x6D1gziVcYqZbWfStZZMqBbF\njxS7sHptGlRg6dBQ0q/fpOfcHRy/cP2x1jmZcZIB6wZwKesS8zrM48mqTxqcVAhjSLELm9DU24MV\nkWHk5Gl6z0vg+9Qrj3T7gxcPMmjdIHLzc1nceTFNKjYppKRCFJwUu7AZPpXLsnpEGK6Odjw7fyc7\nTlx8qNvtPrebIRuG4OrgypIuS6jvUb+QkwpRMFLswqZ4e5Zi9YjmVHF3YdDiPWw4eP6+2286s4mR\n8SOpXLoycZ3jqF62ehElFeLxSbELm/OEmzMro8LwqVSWkR8nsSrxx3tu9/nxzxmzeQwNPBoQ2zmW\niqUqFnFSIR6PIcWulFqklLqglDpgxHpCFLZyro4sGxZKizqevLL6e2K2nbzr60sOLuGN794g5IkQ\nFnRcgJuTm0lJhXh0Rh2xxwKdDVpLiCJRysmemIHBdPV7gn99dYhpG46Qn5/PrL2zeC/xPTrU6MDs\ndrNxdXA1O6oQj8SQkQJa661KKW8j1hKiKDnZ2/Hhs00o67yfj749ypafozl9K55n6j7DhGYTsLMY\nP4pAiMJWZLNilFKRQCRA9eryBJQoPuwsisndGnAgZzanb+6gmqUrrzWVUhclV5E9eaq1nq+1DtZa\nB3t5eRXVboV4oKzcLP62+W/8cHMHYe4RpBx8ksilSWTdyjM7mhCPRa6KETYt42YGkf+NJOGnBCaG\nTWR++CtMfcaPbcfS6b9wFxmZxg4PE6IoSLELm3Ux6yJDNgzhwKUDvPfke/So1wOAviHVmf1cE/an\nZtBnfgIXrmWbnFSIR2PU5Y7LgQSgvlIqVSk11Ih1hSgsqddSiVgXwY/XfmR2u9l09O5419e7+FVi\n0aCmnPk5k17RCfz4c6ZJSYV4dIYUu9b6Wa11Ja21g9a6qtZ6oRHrClEYjl0+RsS6CDJuZrCg4wKa\nV25+z+1a1vVk2bBQrmTm0GPuDo6cv1bESYV4PHIqRtiUfen7GLR+EACxnWNp7NX4vtsHVndn1Ygw\nlILe8xLYe+ZyEaQUomCk2IXN2PHTDob/dzhuTm4s6bKEuu51H+p29SqWYfWI5ri7OtBvwS62HUsv\n5KRCFIwUu7AJ/z39X0ZtGkW1MtVY0mUJVctUfaTbV/NwZdWI5nh7lmJI7B6+3n+ukJIKUXBS7MLq\nrT66mpe3vIyfpx+LOy/G08XzsdbxKuPEp5HNaFy1HC98spflu88YnFQIY0ixC6u2cP9CJiVMokWV\nFszrMI+yjmULtJ6biwNLh4byZD0vxv97P3M3nzAoqRDGkWIXVklrzQeJHzBj7wy61OzCrDazcLF3\nMWRtF0c7FkQE0y2gMu+sP8zbXx9Ca23I2kIYochmxQhRVPLy85i8czL/PvZv+tTvw2uhr2FRxh7D\nONhZmN47ADcXB+ZtPcmVzBzeesYPO4sydD9CPA4pdmFVbuXdYty2cWz8YSOR/pG8EPACShVO2Vos\niknhvpRzcWDWN8e5mp3DjL4BONnL8DBhLil2YTUyczJ58dsX2XluJ682fZUBPgMKfZ9KKcZ0rI+b\nqyNvrk3hWmwi8wYEUcpJ/msJ88g5dmEVrmRfYdh/h7Hn/B7+1eJfRVLqvzW0ZU3e79WYhJOXeC5m\nF5dv3CrS/QvxW1LsosRLu5HGoPWDOPLzET5o/QHd6nQzJUePoKpE9w/i0Lmr9J6XwPkMGR4mzCHF\nLkq0H67+QMS6CM5nnie6QzRtq7c1NU8Hn4rEDQ7hXEY2Pebu4NTFG6bmEbZJil2UWId/PkzEugiy\ncrNY2GkhTZ9oanYkAMJql2f58GZk5eTRK3oHB3/KMDuSsDFS7KJE2pu2lyHrh+BgcSC2Syy+5X3N\njnQXv6purIwKw9HOQt/5O9lz+mezIwkbIsUuSpytqVuJ2hhFeZfyLO2ylFputcyOdE91KpRm1cjm\neJVxYsDCXXx7+ILZkYSNkGIXJcpXJ7/ixW9epKZbTeK6xFGpdCWzI91XlXIurIoKo26FMgxfksgX\nyWfNjiRsgBS7KDGWH17O+G3jCagQwKJOi/Bw9jA70kMpX9qJT4aHEuztzksrklmScNrsSMLKSbGL\nYk9rTfS+aN7a9RatqrVibvu5lHYsbXasR1LG2YHYwSG0a1CRCV8cZNamYzJfRhQaKXZRrOXrfN7d\n8y6zk2cTXjuc6a2n42zvbHasx+LsYEd0/yb0aFKVDzYeZfLaFPLzpdyF8Qx53bNSqjMwE7ADYrTW\nU41YV9i2nPwcJu6YyH9O/If+DfvzStNXDB/mVdTs7Sy819MfNxcHFn13ioysHN7p4Y+DXcm+X6J4\nKXCxK6XsgNlAByAV2KOU+o/WOqWgawvblZ2bzStbXmFz6mZGBYwiyj+q0IZ5FTWLRfHGUw1xd3Xg\n/Y1HuZqVw0fPNcHZQYaHCWMYccQeAhzXWp8EUEp9CnQDDC/2zJxMbuXJDA5rdzPvJuO2jSMpLYl/\nhv6Tvg36mh3JcEopRrerS7lSjkz44gCDFiYw4+maOMtkSKvnWqYcjk6FezrRiGKvAvz4m49TgVAD\n1v2DD5I+YMWRFYWxtChm7JU9U/8yla61upodpVANaFaDss72OP57ME9E7zI7jigC37daiH+bnoW6\njyKbLaqUigQiAapXr/5Ya3Ty7kRNt5pGxhLFlL+nP35efmbHKBLdAqqQuT2Ly1m1OFK1l9lxRCGr\nUavwH9dGFPtZoNpvPq76y+fuorWeD8wHCA4OfqxLAZo+0bTYzAMRwkiu9uBapR7Nnn3N7CjCChjx\nVPweoK5SqqZSyhHoC/zHgHWFsB06H5ScXxfGKPARu9Y6Vyn1ArCB25c7LtJaHyxwMiFsic4Hi1zy\nKIxhyDl2rfXXwNdGrCWETcrPgxJ+jb4oPuSRJERxoPPkVIwwjBS7EMVBfh5YpNiFMaTYhSgO5Ihd\nGEiKXYjiQGs5xy4MI48kIYqD/Dy5KkYYRh5JQhQHcipGGEiKXYjiQJ48FQaSYheiOND5co5dGEYe\nSUIUB3IqRhhIil2I4iA/X07FCMNIsQtRHGgZKSCMI48kIYoDLUfswjhS7EIUBzIETBhIHklCFAfy\n5KkwkBS7EGbTWk7FCENJsQthNv3LO0XKEbswiBS7EGbTebd/l3PswiDySBLCbPm/FLsMARMGkUeS\nEGa7c8Qup2KEMQpU7EqpXkqpg0qpfKVUsFGhhLApd47YpdiFMQp6xH4AeAbYakAWIWyTzr/9u5xj\nFwaxL8iNtdaHAJRSxqQRwhbdKXY5YhfGKFCxF7kt78GB1WanEMJY+bm3f5dTMcIgDyx2pVQ88MQ9\nvvRPrfUXD7sjpVQkEAlQvXr1hw54l9IVwKv+491WiOKsciDUaWd2CmEllP71xREFWUSpzcDLWuvE\nh9k+ODhYJyY+1KZCCCF+oZRK0lo/8EIVebZGCCGsTEEvd+yulEoFwoCvlFIbjIklhBDicRX0qpg1\nwBqDsgghhDCAnIoRQggrI8UuhBBWRopdCCGsjBS7EEJYGSl2IYSwMoa8QOmRd6pUOvBDke+44DyB\ni2aHKGJyn62frd1fKLn3uYbW2utBG5lS7CWVUirxYV71ZU3kPls/W7u/YP33WU7FCCGElZFiF0II\nKyPF/mjmmx3ABHKfrZ+t3V+w8vss59iFEMLKyBG7EEJYGSn2x6CU+odSSiulPM3OUtiUUu8ppQ4r\npb5XSq1RSpUzO1NhUUp1VkodUUodV0qNMztPYVNKVVNKfauUSvnlTelfNDtTUVFK2Sml/qeUWmt2\nlsIgxf6IlFLVgI7AGbOzFJGNQCOttT9wFBhvcp5CoZSyA2YDXQAf4FmllI+5qQpdLvAPrbUP0AwY\nZQP3+VcvAofMDlFYpNgf3XTgVcAmnpzQWv9Xa/3Lm3KyE6hqZp5CFAIc11qf1FrfAj4FupmcqVBp\nrc9prff+8udr3C66KuamKnxKqarAX4EYs7MUFin2R6CU6gac1VrvMzuLSYYA68wOUUiqAD/+5uNU\nbKDkfqWU8gYCgV3mJikSM7h9cJZvdpDCUqA32rBG93vzbuA1bp+GsSoP84blSql/cvtH92VFmU0U\nPqVUaeAz4CWt9VWz8xQmpdRTwAWtdZJSqrXZeQqLFPvvaK3b3+vzSik/oCawTykFt09J7FVKhWit\nzxdhRMP92X3+lVJqEPAU0E5b7/WxZ4Fqv/m46i+fs2pKKQdul/oyrfW/zc5TBFoA4UqproAzUFYp\n9bHWur/JuQwl17E/JqXUaSBYa10SBwk9NKVUZ+ADoJXWOt3sPIVFKWXP7SeH23G70PcAz2mtD5oa\nrBCp20coccDPWuuXzM5T1H45Yn9Za/2U2VmMJufYxYN8BJQBNiqlkpVS0WYHKgy/PEH8ArCB208i\nrrTmUv9FC2AA0PaXf9vkX45kRQknR+xCCGFl5IhdCCGsjBS7EEJYGSl2IYSwMlLsQghhZaTYhRDC\nykixCyGElZFiF0IIKyPFLoQQVub/A30q93Uaoz3wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcbbcd03940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def softmargin(f,y):\n",
    "    loss = 1.0 - y * f\n",
    "    return loss * (loss > 0)\n",
    "\n",
    "loss = softmargin\n",
    "with ag.record():    # start recording\n",
    "    theloss = loss(output, thelabel)\n",
    "theloss.backward()   # and compute the gradient\n",
    "\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='Soft margin')\n",
    "plt.plot(output.asnumpy(), output.grad.asnumpy(), label='Gradient')\n",
    "# now compute the loss for y=-1\n",
    "theloss = loss(output, -thelabel)\n",
    "plt.plot(output.asnumpy(), theloss.asnumpy(), label='Soft margin for y=-1')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entropy\n",
    "\n",
    "Sometimes we care about probabilities rather than just labels. In particular, we might want to measure the distance between distributions. For that we need some basics about probabilities, such as the [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)). In a nutshell, the entropy of a random variable is the amount of surprise we encounter each time we sample from it. For instance, the entropy of the constant function is zero, since we already know what's coming. The entropy of a fair coin being tossed is 1 bit. We have no idea what's going to happen (it's a fair coin after all) and there are only two possible outcomes. If we had a biased coin, e.g. one that produces heads with probability 0.9 and tails with probability 0.1, the surprise would be less (after all, most of the time we see a head). Correspondingly its entropy should be lower. On the other hand, a dice with 6 possible outcomes should have a higher degree of surprise. Without further ado, let's define the entropy function:\n",
    "\n",
    "$$H[p] := \\sum_x -p(x) \\log p(x)$$\n",
    "\n",
    "This works well for discrete outcomes. For densities we use\n",
    "\n",
    "$$H[p] := \\int -p(x) \\log p(x) dx$$\n",
    "\n",
    "We can check that for a fair coin the entropy is given by $H[p] = -2 \\cdot 0.5 \\log 0.5 = \\log 2$. Information theorists often measure the information in 'nats' rather than bits. The difference is the base of the logarithm. It's easy to convert: 1 nat is $\\log_2 e \\approx 1.44$ bit. More generally, for a uniform distribution over $N$ outcomes it is $H[p] = \\log N$. One of the fundamental theorems in information theory is that for a distribution $p$, we need at least $H[p]$ nats to encode it. \n",
    "\n",
    "There are a number of useful properties for the entropy that are employed in machine learning: \n",
    "\n",
    "* Often when estimating distributions we want to find the one with the largest entropy that fits the requirements. That's in line with our desire to restrict our estimates as little as possible beyond what we actually observe.\n",
    "* The entropy is a concave function. That is, for two distributions $p$ and $q$, the mixture of both has higher entropy: $H[\\lambda p + (1-\\lambda) q] \\geq \\lambda H[p] + (1-\\lambda) H[q]$. To prove this, simply note that the function $-x \\log x$ is concave. \n",
    "* When we have independent random variables, say $x$ and $y$, then the entropy of the joint distribution is the sum of the individual entropies. This follows simply from the fact that $\\log p(x) q(y) = \\log p(x) + \\log q(y)$. \n",
    "* For dependent random variables the joint entropy is lower than that of the individual terms. This can be seen as follows:\n",
    "\n",
    "$$\\begin{eqnarray}\n",
    "H[p(x,y)] = & \\int -p(x,y) \\log p(x,y) \\\\\n",
    "= & \\int -p(x,y) [\\log p(x) p(y)] dx dy + \\int p(x,y) \\log \\frac{p(x) p(y)}{p(x,y)} dx dy \\\\\n",
    "\\leq & H[p(x)] + H[p(y)] + \\log \\int p(x,y) \\frac{p(x) p(y)}{p(x,y)} dx dy \\\\\n",
    "= & H[p(x)] + H[p(y)]\n",
    "\\end{eqnarray}$$\n",
    "\n",
    "Here the inequality follows from the fact that $\\log x$ is a concave function, hence the expectation of the logarithm is less than the logarithm of the expectation. Intuitively this result is straightforward - if $x$ and $y$ are dependent on each other, then knowing $x$ should tell us some more about $y$. Therefore, the joint entropy of $x$ and $y$ should be lower than the sum of the individual entropies. \n",
    "\n",
    "This leads us to the notion of mutual information. It is given by the difference between joint and and independent entropies, i.e. $I(x,y) := H[p(x)] + H[p(y)] - H[p(x,y)]$. Basically it's the amount of information that we save. For instance, a light switch and a (functioning) light bulb are strongly correlated - knowing one tells us all about the other. The entropy of the joint is 1 bit (if it's on with probability 0.5), but the sum of the entropies of switch and bulb individually is 2 bit. Hence the mutual information is 1 bit. \n",
    "\n",
    "## Kullback Leibler Divergence\n",
    "\n",
    "This brings us to the KL divergence. It measures how close two distributions are. One way of defining such a quantity is to ask how many extra bits one would have to spend to encode data drawn from $p$ when using a code tuned for $q$. If we assume for a fact that it takes $-\\log p(x)$ nat to optimally encode $x$, then the penalty from using the 'wrong' code is given by \n",
    "\n",
    "$$D(p\\|q) = \\sum_x p(x) [\\log p(x) - \\log q(x)]$$\n",
    "\n",
    "For densities the quantity is defined analogously, i.e. $\\int p(x) [\\log p(x) - \\log q(x)] dx$. The first thing to prove is that this is actually a distance. For that we need to show that $D(p\\|q) \\geq 0$ with equality only for $p=q$. To see the latter, simply plug $p=q$ into the definition. To see the former, we rewrite $D$ in the same way as above, using convexity, this time of $-\\log x$. \n",
    "\n",
    "$$D(p\\|q) = \\sum_x -p(x) \\log \\frac{q(x)}{p(x)} \\geq -\\log \\sum_x p(x) \\frac{q(x)}{p(x)} = 0$$\n",
    "\n",
    "As an aside, to see that $H[p]$ can be achieved, indeed, quantize all $p(x)$ into bins of the next largest fraction of $2$, e.g. $0.2$ goes into the bin of $\\frac{1}{4}$. It is clear that the sum over those bins is no smaller than $1$ and no larger than $2$. Moreover, we can arrange them into a tree, where at level $l$ the bins are of size $2^{1-l}$. Then we simply index these bins according to their position of the tree. Each $x$ will require $\\lceil \\log_2 p(x) \\rceil$ bits (whatever is left over, we simply discard). In sum this is no more than $\\log_2 H[p] + 1$. To tighten the bound, simply send $N$ symbols. Since they can be encoded using at most $N \\log_2 H[p] + 1$ bit, the code becomes increasingly efficient with only $1/N$ waste. This proves that such a code can be found. That it's impossible to do any better is a consequence of $D(p\\|q) \\geq 0$.  \n",
    "Note that our construction relied on very long codes for efficiency. This is a real problem in practice. [Turbo codes (https://en.wikipedia.org/wiki/Turbo_code) are one of the techniques to address this, e.g. for mobile communications. \n",
    "\n",
    "After this long detour, let's finally get to the KL divergence as a loss function. It generalizes the multiclass softmax as follows: instead of having just a single possible true class, it uses a probability distribution as reference. That is\n",
    "\n",
    "$$\\log(f,y) = \\sum_i y_i (log(y_i) - f_i)$$\n",
    "\n",
    "Here $f_i$ is assume to be a probability distribution (or we can set a flag to transform the output into one beforehand). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ 0.03703965]\n",
      "<NDArray 1 @cpu(0)>\n",
      "\n",
      "[  1.28348230e-08]\n",
      "<NDArray 1 @cpu(0)>\n",
      "\n",
      "[ 0.16921175]\n",
      "<NDArray 1 @cpu(0)>\n",
      "\n",
      "[ 1.69211745]\n",
      "<NDArray 1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "loss = gluon.loss.KLDivLoss()\n",
    "\n",
    "# generate some random probability distribution\n",
    "f = nd.random_normal(shape=(1,10))\n",
    "p = nd.exp(f)\n",
    "p = p / nd.sum(p)\n",
    "\n",
    "# generate some target distribution\n",
    "y = nd.random_normal(shape=(1,10))\n",
    "y = nd.exp(y)\n",
    "y = y / nd.sum(y)\n",
    "\n",
    "z = nd.zeros_like(y)\n",
    "z[0,3] = 1\n",
    "\n",
    "# distance between our estimate p and the 'true' distribution y\n",
    "print(loss(nd.log(p), y))\n",
    "# distance to itself - should be zero\n",
    "print(loss(nd.log(p), p))\n",
    "# equivalent of logistic loss with class 3 up to normalization over domain, i.e. 1/10\n",
    "# note that this is VERY DIFFERENT from information theory but a traditional choice\n",
    "# in deep learning\n",
    "print(loss(nd.log(p), z))\n",
    "print(-nd.log(p[0,3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "[The learning environement](../chapter02_supervised-learning/environment.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
